Video Screencast Help
Symantec Appoints Michael A. Brown CEO. Learn more.

Stream scanning vs File Scanning issues

Created: 06 Oct 2011 | 12 comments

We had received a requirement to scan files with 1.2 GBytes on size, and we are looking on stream scan process.

We have been testing our Symantec Scanning Server Engine with the following command line which call a wrapper class that wait for any key and then call the  sample program JavaAPICheck.java with the parameters showed, we are using JVisualVM to monitor the Memory,etc.

 

 
We are experienced several issues:
 
First.- Our JVisualVM show a consumption on memory on the client side that we didn't expect under streambased virus scan case. i.e if the file is 253MB size the amount of memory show around 300MB used on the client side.
 
Second.- Connections error messages with a test file that has a virus, following the error messages:
Problem encountered! Scanning Failed!! ERROR_ON_SERVER_WRITE
Problem encountered! Scanning Failed!! ERROR_SOCKET_COMMUNICATION
 
 
Any help is greatly appreciated.
 
 
Detail of 3 tests using 2 differents files:
 
Test # 1
 
E:\POC\symantec_test>run.bat
 
E:\POC\symantec_test>java -Xmx1500M -Xms512M -cp SymJavaAPI.jar;target\classes;.\  org.ocmm.symantec.App -file:..\virus\AgencyApplicationDownload.zip
-streambased:1 -streamFileLocal:1 -server:209.222.132.148:1344 -policy:scan -output:..\virus\AgencyApplicationDownload.zip-out -originalname:..\virus\
AgencyApplicationDownload.zip
Click <enter> key to continue...
 
Arguments [
-file:..\virus\AgencyApplicationDownload.zip
-streambased:1
-streamFileLocal:1
-server:209.222.132.148:1344
-policy:scan
-output:..\virus\AgencyApplicationDownload.zip-out
-originalname:..\virus\AgencyApplicationDownload.zip
 ]
Problem encountered! Scanning Failed!! ERROR_ON_SERVER_WRITE
 
Test # 2
E:\POC\symantec_test>run.bat
 
E:\POC\symantec_test>java -Xmx1500M -Xms512M -cp SymJavaAPI.jar;target\classes;.\  org.ocmm.symantec.App -file:..\file253MB.pdf -streambased:1 -stream
FileLocal:1 -server:209.222.132.148:1344 -policy:scan -output:..\file253MB.pdf-out -originalname:..\file253MB.pdf
Click <enter> key to continue...
 
Arguments [
-file:..\file253MB.pdf
-streambased:1
-streamFileLocal:1
-server:209.222.132.148:1344
-policy:scan
-output:..\file253MB.pdf-out
-originalname:..\file253MB.pdf
 ]
----------------------------------------------------------------------
Scanning file ........................................................
----------------------------------------------------------------------
Results ..............................................................
----------------------------------------------------------------------
File Scanned            : ..\file253MB.pdf
Scan Policy             : SCAN
File Status             : CLEAN
Total Infection         : 0
Virus Def Date          : Fri Sep 10 00:00:00 EDT 2010
Virus Def Revision No   : 003
Scan Engine IP          : 209.222.132.148
Scan Engine Port        : 1344
Scan Engine Port        : Able to connect
 
Test # 3
 
E:\POC\symantec_test>run.bat
 
E:\POC\symantec_test>java -Xmx1500M -Xms512M -cp SymJavaAPI.jar;target\classes;.\  org.ocmm.symantec.App -file:..\virus\AgencyApplicationDownload.zip
-streambased:1 -streamFileLocal:1 -server:209.222.132.148:1344 -policy:scan -output:..\virus\AgencyApplicationDownload.zip-out -originalname:..\virus\
AgencyApplicationDownload.zip
Click <enter> key to continue...
 
Arguments [
-file:..\virus\AgencyApplicationDownload.zip
-streambased:1
-streamFileLocal:1
-server:209.222.132.148:1344
-policy:scan
-output:..\virus\AgencyApplicationDownload.zip-out
-originalname:..\virus\AgencyApplicationDownload.zip
 ]
Problem encountered! Scanning Failed!! ERROR_SOCKET_COMMUNICATION

Comments 12 CommentsJump to latest comment

TSE-JDavis's picture

These are example files you are using. They are not designed for production usage.

You should be using ssecls, which is our compiled command-line scanning tool.

The expansion in memory is to be expected. You sent us a PDF file which is a container. We have to expand that container to scan all of the pieces inside of it.

OswaldoAtGdit's picture

TSE-JDavis,

Thanks for your answer but we are interested in the java API not in the commnad line version, we are using the command line for other use cases.

TSE-JDavis's picture

You will want to build on the example files we have created and flesh out the code providing for more memory usage and adjusting timeouts for the sizes of files you will be scanning.

OswaldoAtGdit's picture

TSE-JDavis,

thanks for your replies.

 

I wounder which streaming scanning will be more appropriate for large size file ( over 1.2 GBytes).

1.- I understand using the compiled command line option as you suggested without the java api.

2.- Using java api streaming chunks of the file to the server.

3.- Using java api to pass the server:dynamic file path location of the file in a shared directory.

Considering efficiency in resources.

 

thanks in advance

Benc_Smith's picture

Hi Oswald,

For test #1 it sounds like we could not write to the output file for some reason.  Do you consistently get this Error On Server Write error, or was this just a one off? 

For test #3 how large is the .zip file in question, is it fairly large?  Is Scan Engine under a lot of load, in terms of a concurrent requests?  If the .zip file is fairly large you might be running into the Socket communication error because our Java API is hitting our default readWriteTime timeout parameter.  By default it is only 3 seconds.  You can try increasing this parameter to a larger value, for example 60 seconds.  Then confirm if the issue is still occurring.  You can set the readWriteTime parameter when you call the createScanEngine function.  The parameter is set when you call createScanEngine. 

Regarding the memory usage of the client, I believe the client typically uses a buffer in a loop to pass/stream the file to the scanner.  Which is what our example program does, but our example program creates a new buffer array every time it goes through the, in this case a do while, loop.  If you are doing something similar, this might be where the memory consumption is coming from.  I do not believe the memory allocated each time the array of bytes is created is cleared until the function is finished.  So this could be accounting for the memory usage you are seeing on the client.  If this is case, this is not on our API, it would be on the connector you wrote.  Just do not create a "new" array of bytes every time you go through the while loop.

OswaldoAtGdit's picture

Benc Smith,

Thanks for your replies. Our Test 1 and 3 are using the same resources and same zip file and the error are interchangeable.

I do appreciate the tip about the array buffer, I will test it.

Could you suggest me what will be the best way, using the java api to be able to process large files above 1.2 GBytes.

1.- Streaming chunks pieces one at the time.

2.- Perhaps communicate using specific symantec protocol from java api to pass the server:dynamic-file-path in a share directory.

thanks.

Anil143's picture

Hello,

We are working on .Net website application. Using the application, users can upload documents and those attachments to be scanned during upload to the server (could be from some temp folder on file server).

For achieving this we are planning to use Symantec Scan Engine 5.2 for scanning the file. We observed different classes/methods available in .NET SDK and we are confused with the Utility (StreamScanFile) and StreamScanRequest options.

 As per the help files provided,

Utility - Abstracts the local stream file scanning mehcanism. It can be used when the file to be scanned is accessible only to the connector and not to the Scan engine.

StreamScanRequest  - Abstracts the stream file scanning mehcanism. This is used to Scan file which is not accessible to Scan engine. 

Our understanding is both of these classes will be used when ScanEngine does not have access to the file. Utility class will be used to scan the file when the connector (application) has access to the file. Does that mean the other one (StreamScanRequest) is useful if the connector does not have access on the file? If so then how it can read the file and send the bytes to scan engine? In which scenario we will go with the StreamScanRequest class?
 
 
Please let us know, which is the best way to go with by considering performance and security?
 
Thanks,
Anil.
TSE-JDavis's picture

The two options are the difference between streaming the file to Scan Engine, or telling Scan Engine the location of the file on a shared drive that Scan Engine will be able to access (example: \\webserver\scanenginetemp\testfile.txt).

Anil143's picture

Thanks Dave for your immediate response. But as per the help file provided in .NET SDK,  Utility class method can be used when the file to be scanned is accessible only to the connector and not to the Scan engine. What does the connector means? Is it calling application or Scan Engine Server? Our understanding was application. Please confirm.

Thanks, Anil.

Benc_Smith's picture

Hi Anil,

Your application, which is using our .NET API, is the "connector". 

Thanks,
Ben

Anil143's picture

Hi,

We are working on a .Net application which uses Symantec Scan engine .net API for file scan. I tried using the Utility option. When we called the API method, we did not get any error. But the scan result is showing the socket error and below is the full information.

 
File Status - CLEAN
Total Infections - 0
Definition Date: 1/1/0001 12:00:00 AM
 
and scanResult.connTriesInfo.problemEncountered as ERR_SOCKET_ERROR. i am not sure whether the scan is successful or not as it shows connection error.
 
Please suggest what could be the issue. 

Regards,

Anil.

Anil143's picture

Hi,

I identified the above issue (ERR_SOCKET_ERROR) issue and is because of passing incorrect port number. Now I am getting ERR_INITIALIZING_STREAM_REQUEST error while calling the below method. Can you please suggest what could be the issue.

objStreamScanRequest.Start(objFileScanEntity.fileForScan, objFileScanEntity.scanFileOriginalName);

 
Thanks,
Anil.