Protection Engine for Cloud Services

 View Only
  • 1.  StreamScanRequest is not streaming in realtime

    Posted Feb 26, 2014 11:35 PM

    I'm uploading large pdf file through http, using a multipart parser to get a stream, then pass the stream to StreamScanRequest, and try to save the result to disk.  

    If I let scanobj.Finish(filestream) to save the file, it's slow, speed down from regular upload of 50M/sec to about 200-300K/sec.  

    To improve performance, I tried to do parallel steps.  In the same loop that read incoming stream to ByteArray, I first scanobj.Send(data), then save the same array to file stream.  After the loop, call scanobj.Finish(null) and fileStream.close.  It works.  As the upload continue, I can see the file growing on disk and get the correct scan result eventually.  

    The problem is, there's a long delay from the file finishing uploading to getting the scan result.  I timed it, the delay is almost exactly the same as reading the same local file and send it for scanning.  It looks as if StreamScanRequest took in a stream, but waited until upload finish before sending it to the scan server.

    Did I miss something?  Can I finish saving local and sending to scan server at the same time, and get the scan result quicker?  Thanks

     



  • 2.  RE: StreamScanRequest is not streaming in realtime

    Posted Mar 04, 2014 10:05 AM

    Are you using RPC or ICAP?  Only ICAP is capable of data trickling, which I think is what you're describing here



  • 3.  RE: StreamScanRequest is not streaming in realtime

    Posted Mar 04, 2014 04:49 PM

    Thank you so much for following up on this.  I'm not exactly sure what I used, here's a code snippet that I adapted from the sample code.   

    StreamScanRequest scan = requestManagerObj.CreateStreamScanRequest(Policy.DEFAULT);
    scan.Start(fileName, fileName);
    byte[] buffer = new byte[16 * 1024];
     
    using (FileStream fileStream = ... ) {
        int read = 0;
     
        while ((read = stream.Read(buffer, 0, buffer.Length)) > 0) {
            scan.Send(read < buffer.Length ? buffer.Take(read).ToArray() : buffer);
            fileStream.Write(buffer, 0, read);
        }
        ScanResult result = scan.Finish(fileStream);
    }
     
    I guess it's not since the scan won't start until after the file stream finished.
     
    Is there a C# ICAP client example?  The latest I can find is back in 2007.  While I manage to make it work, it's extrememly slow.  A 50M pdf takes 1:37, while ssecls.exe took 5 seconds.  Is there a more uptodate example?  Thanks.
     
    http://vaibhavkulkarni.wordpress.com/2007/11/19/a-icap-client-code-in-c-to-virus-scan-a-file-using-symantec-scan-server/
     
     
    P.S.
     
    After some further reading, it seems StreamScan is using ICAP underneath, but I don't have control over the streaming process.  The send for scanning process won't start until upload finishes, even though I try to pass the stream around.
     
    Speed wise, the low level ICAP client has the same extreme slow speed as scan.Finish(fileStream).  If I use scan.Finish(Stream.null), it gest a little faster.  But everything is magintude below ssecls.exe
     
    I was hoping there's a way to stream Web=>Service=>Scanner=>FileSave synchronously, at reasonable speed, so the UI progress keep up with all the backend process.
     
    Thanks.


  • 4.  RE: StreamScanRequest is not streaming in realtime

    Posted Mar 06, 2014 06:03 AM

    TBH the forum isn't the best place for this lowlevel stuff.  Symantec offer the API, tell you what functions do what.  But anything other than that is out of Symantec's remit and is down to your own code.  The engine accepts what's given to it and its the engine that dictates what protocol it scans in.  Make sure you're scanning in ICAP and not RPC.



  • 5.  RE: StreamScanRequest is not streaming in realtime

    Posted Mar 06, 2014 11:22 AM

    Thanks, maybe a more general question.  According to this

    "Scan Engine includes an SDK for client-side ICAP to allow C++, Java and C# (for .NET integrations) to quickly link Symantec Scan Engine with your own application."

    The StreamScanRequest in C# sample should or could use ICAP.  How can I tell which one it's using?  And how can I config it and force trickle?  The SDK developer guide go right into raw ICAP.

     



  • 6.  RE: StreamScanRequest is not streaming in realtime

    Posted Mar 06, 2014 11:40 AM

    If you're using that SDK I think it overrides the setting on scan engine, but inside the scan engine there's a setting that shows you what its using.

     

    Here's the feature comparisons http://www.symantec.com/docs/TECH131848

    http://www.symantec.com/security_response/glossary/define.jsp?letter=d&word=data-trickle

    http://www.symantec.com/docs/TECH96867

    To enable data trickling

    1 In the console on the primary navigation bar, click Configuration.
    2 In the sidebar under Views, click Protocol.
    3 In the content area under Trickle, check Enable trickle.
    Data trickling is disabled by default.


  • 7.  RE: StreamScanRequest is not streaming in realtime

    Posted Mar 06, 2014 11:47 AM

    There is a good chuck of text in the manual about data trickling, you might want to have a look at that (that last bit in my previous post was a copypaste)

    ftp://ftp.symantec.com/public/english_us_canada/products/symantec_scan_engine/5.1/manuals/sse51_imp_guide.pdf 



  • 8.  RE: StreamScanRequest is not streaming in realtime

    Posted Mar 06, 2014 06:38 PM

    Our servers are controlled by a different group.  I finally manage to setup my own trial copy, but there's no settings for trickling.  Also, all documentation mentioning data trickling are 5.1.  You can't even find the word trick/trickle/trickling in any of the version 7 imp/dev guide.  Any ideas?  Thanks.

    SPE_ICAP.jpg



  • 9.  RE: StreamScanRequest is not streaming in realtime

    Posted Mar 07, 2014 03:59 AM

    Hmm, either its a commandline setting or its a feature thats been removed.  You should contact tech support to verify.