A proposal to support keyword scanning within e... | MDaemon Technologies Community Forum

A proposal to support keyword scanning within email attachments in MDaemon's Content Filter.


  • Excuse me, does MDaemon plan to support scanning the content of email attachments in its content filter?

    For example, I would like to filter emails whose Word document attachments contain the word 'invoice', and have those emails moved to the bad queue.

     



  • MDaemon does not directly support searching the content of email attachments, but it can be accomplished using a content filter rule and a powershell script. If you'd like an example of a script that could be used, let me know.

    If you'd rather have something built into the product but want it now our SecurityGateway product does support searching the content of email attachments for keywords, https://mdaemon.com/pages/security-gateway

     


  • Hi Arron,

    Please let me know how to implement this using PowerShell.

    Thank you!


  • You'll need to create two content filter rules similar to the following:

    [Rule004]
    RuleName=DOC Attachment Scan
    Enable=Yes
    ThisRuleCondition=All
    ProcessQueue=LOCAL
    Condition01=body|attachment name|AND|*.doc*
    Action01=run a program|"-1,0,1","C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -ExecutionPolicy Bypass -File "C:\MDaemon\App\ScanWordAttach.ps1" "$MESSAGEFILENAME$""
    [Rule005]
    RuleName=Attachment Action 
    Enable=Yes
    ThisRuleCondition=All
    ProcessQueue=LOCAL
    Condition01=body|process exit code|AND|=|1|
    Action01=move to bad Msg|

    Notice the rules only apply to the LOCAL queue.  This means that only messages being delivered to local accounts will cause the rules to processed.  You'll need to save the powershell script below to disk at C:\MDaemon\App\ScanWordAttach.ps1, or adjust the rules accordingly.  You can adjust the words that it searches for by creating a C:\MDaemon\App\ScanKeywords.txt file on disk. Blank lines and lines
    starting with # are ignored. One word per line.  Additional details are in the comments of the script, please read them all.  The script is designed to work on modern docx files. it does not work for PDFs.

    <#
        ScanWordAttach.ps1
        ------------------
        MDaemon Content Filter helper. Scans document attachments of ONE queued
        message for one or more keywords/phrases and signals the result via its
        EXIT CODE:
    
            0  = no match (or no scannable attachment)  -> let the message pass
            1  = MATCH                                  -> Content Filter moves it to the bad queue
            2  = could not scan (error)                 -> fail-open, do NOT quarantine
    
        Called from a Content Filter "Run a program" action as:
            C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -ExecutionPolicy Bypass -File "C:\MDaemon\App\ScanWordAttach.ps1" "$MESSAGEFILENAME$"
    
        KEYWORDS (multiple supported) - two ways, combined if both are present:
          1. Keyword file (recommended): one term per line in $KeywordsFile
             (default C:\MDaemon\App\ScanKeywords.txt). Blank lines and lines
             starting with # are ignored. Edit the list without touching the rule.
          2. Inline: -Keywords "invoice;payment;remittance advice"  (';' , ',' or '|')
    
        Match mode:
          * Default = ANY term present -> match.
          * -MatchAll = every term must appear somewhere in the message's attachments.
          * -WholeWord = match whole words only (e.g. 'invoice' not 'invoices').
    
        SELF-CONTAINED - no third-party tools. Windows / .NET only.
        Coverage: Office Open XML (docx/xlsx/pptx) exact; legacy binary (doc/xls/ppt/rtf)
        best-effort; PDF not scanned. Windows PowerShell 5.1 compatible.
    #>
    param(
        [Parameter(Mandatory = $true)] [string] $MessagePath,
        [string] $Keywords     = 'invoice',
        [string] $KeywordsFile = 'C:\MDaemon\App\ScanKeywords.txt',
        [switch] $MatchAll,
        [switch] $WholeWord,
        [string] $LogFile      = 'C:\MDaemon\Logs\ScanWordAttach.log'
    )
    
    $EXIT_NOMATCH = 0
    $EXIT_MATCH   = 1
    $EXIT_ERROR   = 2
    
    $code = $EXIT_ERROR
    $temp = New-Object System.Collections.Generic.List[string]
    
    function Write-Log([string]$m) {
        try { Add-Content -LiteralPath $LogFile -Value ((Get-Date -Format 'yyyy-MM-dd HH:mm:ss') + '  ' + $m) -Encoding UTF8 } catch { }
    }
    
    # Exact Open XML text extraction (docx/xlsx/pptx). $entryRegex selects the text-bearing parts.
    function Get-OpenXmlText([string]$path, [string]$entryRegex) {
        $sb  = New-Object System.Text.StringBuilder
        $zip = [System.IO.Compression.ZipFile]::OpenRead($path)
        try {
            foreach ($e in $zip.Entries) {
                if ($e.FullName -match $entryRegex) {
                    $r = New-Object System.IO.StreamReader($e.Open())
                    try { $x = $r.ReadToEnd() } finally { $r.Dispose() }
                    $x = $x -replace '<w:tab\b[^>]*/>', "`t"
                    $x = $x -replace '</(w:p|a:p)>', "`n"
                    $x = [regex]::Replace($x, '<[^>]+>', '')
                    [void]$sb.Append([System.Net.WebUtility]::HtmlDecode($x))
                    [void]$sb.Append("`n")
                }
            }
        } finally { $zip.Dispose() }
        return $sb.ToString()
    }
    
    # Best-effort scrape for legacy binary formats (no native API without Office).
    function Get-LegacyDocText([string]$path) {
        $b = [System.IO.File]::ReadAllBytes($path)
        return ([System.Text.Encoding]::GetEncoding(1252).GetString($b) + "`n" + [System.Text.Encoding]::Unicode.GetString($b))
    }
    
    # Extract text from a decoded attachment based on its detected kind.
    function Get-AttachmentText([string]$file, [string]$kind) {
        switch ($kind) {
            'word'  { return (Get-OpenXmlText $file '^word/(document|header\d*|footer\d*|footnotes|endnotes)\.xml$') }
            'excel' { return (Get-OpenXmlText $file '^xl/(sharedStrings\.xml|worksheets/sheet\d+\.xml)$') }
            'ppt'   { return (Get-OpenXmlText $file '^ppt/(slides/slide\d+|notesSlides/notesSlide\d+)\.xml$') }
            default { return (Get-LegacyDocText $file) }   # legacy binary / rtf
        }
    }
    
    try {
        Add-Type -AssemblyName System.IO.Compression.FileSystem
    
        # ---- Build the term list (inline + file), then precompile a regex per term ----
        $terms = @()
        if ($Keywords) { $terms += ($Keywords -split '[;,|]') }
        if ($KeywordsFile -and (Test-Path -LiteralPath $KeywordsFile)) { $terms += (Get-Content -LiteralPath $KeywordsFile) }
        $terms = $terms | ForEach-Object { $_.Trim() } | Where-Object { $_ -ne '' -and -not $_.StartsWith('#') } | Select-Object -Unique
    
        $rx = @()
        foreach ($t in $terms) {
            $pat = [regex]::Escape($t)
            if ($WholeWord) { $pat = '\b' + $pat + '\b' }
            $rx += New-Object PSObject -Property @{ Term = $t; Rx = (New-Object System.Text.RegularExpressions.Regex($pat, [System.Text.RegularExpressions.RegexOptions]::IgnoreCase)) }
        }
    
        if (-not (Test-Path -LiteralPath $MessagePath)) {
            Write-Log "ERROR message file not found: $MessagePath"
        }
        elseif ($rx.Count -eq 0) {
            Write-Log "WARN no keywords configured - nothing to match"
            $code = $EXIT_NOMATCH
        }
        else {
            # Read raw message using Latin1 (28591) so byte offsets/base64 lines stay intact.
            $raw   = [System.IO.File]::ReadAllText($MessagePath, [System.Text.Encoding]::GetEncoding(28591))
            $lines = $raw -split "\r?\n"
    
            # Gather every MIME boundary declared anywhere (handles nested multiparts).
            $boundaries = @{}
            foreach ($m in [regex]::Matches($raw, '(?im)boundary\s*=\s*"?([^";\r\n]+)"?')) {
                $boundaries[$m.Groups[1].Value.Trim()] = $true
            }
            function Is-BoundaryLine([string]$ln) {
                $t = $ln.TrimEnd()
                if (-not $t.StartsWith('--')) { return $false }
                $b = $t.Substring(2)
                if ($b.EndsWith('--')) { $b = $b.Substring(0, $b.Length - 2) }
                return $boundaries.ContainsKey($b.Trim())
            }
    
            # Split the message into MIME segments at boundary lines.
            $segments = New-Object System.Collections.Generic.List[object]
            $cur = New-Object System.Collections.Generic.List[string]
            $started = $false
            foreach ($ln in $lines) {
                if (Is-BoundaryLine $ln) {
                    if ($started) { $segments.Add($cur.ToArray()) }
                    $cur = New-Object System.Collections.Generic.List[string]; $started = $true; continue
                }
                if ($started) { $cur.Add($ln) }
            }
            if ($started -and $cur.Count -gt 0) { $segments.Add($cur.ToArray()) }
    
            $extWord   = '.docx', '.docm', '.dotx', '.dotm'
            $extExcel  = '.xlsx', '.xlsm', '.xltx', '.xltm'
            $extPpt    = '.pptx', '.potx', '.ppsx', '.pptm'
            $extLegacy = '.doc', '.dot', '.xls', '.xlt', '.ppt', '.pot', '.rtf'
    
            $matched  = $false
            $allText  = New-Object System.Text.StringBuilder   # accumulated only for -MatchAll
    
            foreach ($seg in $segments) {
                # Separate headers from body at the first blank line; unfold folded headers.
                $hdr = New-Object System.Collections.Generic.List[string]
                $bodyStart = -1
                for ($i = 0; $i -lt $seg.Count; $i++) {
                    if ($seg[$i] -eq '') { $bodyStart = $i + 1; break }
                    if ($seg[$i] -match '^[ \t]' -and $hdr.Count -gt 0) {
                        $hdr[$hdr.Count - 1] = $hdr[$hdr.Count - 1] + ' ' + $seg[$i].Trim()
                    } else { $hdr.Add($seg[$i]) }
                }
                if ($bodyStart -lt 0 -or $bodyStart -ge $seg.Count) { continue }
                $H = ($hdr -join "`n")
    
                $cte = '';   if ($H -match '(?im)^Content-Transfer-Encoding:\s*([^\r\n]+)') { $cte = $Matches[1].Trim().ToLower() }
                $ctype = ''; if ($H -match '(?im)^Content-Type:\s*([^\r\n;]+)') { $ctype = $Matches[1].Trim().ToLower() }
                $fname = ''; if ($H -match '(?im)name\*?\s*=\s*"?([^";\r\n]+)"?') { $fname = $Matches[1].Trim() }
                $ext = '';   if ($fname) { $ext = [System.IO.Path]::GetExtension($fname).ToLower() }
    
                $kind = $null
                if     (($ext -in $extWord)  -or ($ctype -like '*wordprocessingml*'))  { $kind = 'word' }
                elseif (($ext -in $extExcel) -or ($ctype -like '*spreadsheetml*'))     { $kind = 'excel' }
                elseif (($ext -in $extPpt)   -or ($ctype -like '*presentationml*'))    { $kind = 'ppt' }
                elseif (($ext -in $extLegacy) -or ($ctype -like '*msword*') -or ($ctype -like '*ms-excel*') -or ($ctype -like '*ms-powerpoint*') -or ($ctype -like '*rtf*')) { $kind = 'legacy' }
                if (-not $kind) { continue }
    
                if ($cte -ne 'base64') { Write-Log "WARN non-base64 part '$fname' (cte=$cte) - skipped"; continue }
    
                $b64 = -join ($seg[$bodyStart..($seg.Count - 1)] | ForEach-Object { $_.Trim() })
                if ([string]::IsNullOrWhiteSpace($b64)) { continue }
                try { $bytes = [Convert]::FromBase64String($b64) } catch { Write-Log "WARN base64 decode failed '$fname'"; continue }
    
                if (-not $ext) { $ext = '.bin' }
                $tmp = [System.IO.Path]::Combine($env:TEMP, ('mdcf_' + [Guid]::NewGuid().ToString('N') + $ext))
                [System.IO.File]::WriteAllBytes($tmp, $bytes)
                $temp.Add($tmp)
    
                $text = ''
                try { $text = Get-AttachmentText $tmp $kind }
                catch { Write-Log "WARN extract failed '$fname': $($_.Exception.Message)"; continue }
                if (-not $text) { continue }
    
                if ($MatchAll) {
                    [void]$allText.Append($text); [void]$allText.Append("`n")
                }
                else {
                    # ANY term in this attachment -> match, stop scanning.
                    foreach ($r in $rx) {
                        if ($r.Rx.IsMatch($text)) {
                            Write-Log "MATCH term '$($r.Term)' in attachment '$fname' ($kind)  msg=$MessagePath"
                            $matched = $true
                            break
                        }
                    }
                    if ($matched) { break }
                }
            }
    
            if ($MatchAll -and -not $matched) {
                $combined = $allText.ToString()
                $missing = @()
                foreach ($r in $rx) { if (-not $r.Rx.IsMatch($combined)) { $missing += $r.Term } }
                if ($missing.Count -eq 0) {
                    Write-Log "MATCH all terms present ($($terms -join ', '))  msg=$MessagePath"
                    $matched = $true
                } else {
                    Write-Log "NOMATCH (all-mode) missing: $($missing -join ', ')  msg=$MessagePath"
                }
            }
    
            if ($matched) { $code = $EXIT_MATCH } else { $code = $EXIT_NOMATCH }
        }
    }
    catch {
        Write-Log "ERROR $($_.Exception.Message)"
        $code = $EXIT_ERROR
    }
    finally {
        foreach ($f in $temp) { try { Remove-Item -LiteralPath $f -Force -ErrorAction SilentlyContinue } catch { } }
    }
    
    exit $code
    

    We have done limited testing on this script. Please test it thoroughly before implenting in production. The script is not supported by MDaemon Technologies and is yours to customize as you see fit. Use at your own risk.

     


  • One other thing I should add, you can implement support for old style windows files and pdfs, but it requires the use of third party tools.


  • Hi Arron,

    Thank you for your reply, and have a good day.

    Is it that outbound emails cannot be scanned, and that only internal (domain) emails can be scanned and moved to the bad queue?


  • The outbound queue can be scanned.  Change the content filter rules to run in the local and remote queue and it should also scan outbound emails.


Please login to reply to this topic!