Data Loss Prevention

Good Morning,

 

We're in the process of running network discover scans on several of the file shares located within our network. About half of our scans end in error:

 

12/8/16 10:17:25 PM	WARNING	Failed to read \\xxxx; error: Unknown error. See the log files for details.
12/8/16 10:17:25 PM	WARNING	Failed to complete Share: \\xxx; error: Unknown error. See the log files for details.
12/8/16 10:17:25 PM	INFO	Scan finished

 

We initially thought that this was due to a permissions issue, however, the DLP service account used for the scanning has domain admin access. I've looked through the logs on both the Enforce server and the Network Discover server used for scanning and have turned up with nothing. I will attempt to run an incremental scan tonight to see if it will complete.

 

Has anyone ran into this problem before?

Hey,

Unknown error can mean many different things.

Try map the shares you're targetting while RDP'd into the Discover servers with the service account. If it doesn't work, it could be firewall/network related or try using the IP of the servers instead of DNS names.

Dean

Hi Anthony,

Try "net use \\Server\shared\folder\ /user:service-account@domain.local passwd"
See if system error xxx has occurred.

Not sure if this is related but last year we set a new baseline GPO, this year we tried Discovery scanning and it was failing in seconds.  To fix this on individual servers we disabled "Network Access: Do not allow storage of passwords and credentials for network authentication".

Hope thats of interest.

Cheers
Dan

 

So, we turned on Advanced Trace Logging on the detection (Network Discover) server and received a few different errors:

1).

Dec 12, 2016 7:00:01 AM com.vontu.discover.crawler.framework.AsynchronousItemHandler acquireMemoryResource

INFO: Interrupted while acquiring memory resource Fetch com.vontu.filesystemcrawler.FileItem@799accaa

•  

                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)

                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2173)

                at com.vontu.resourcemanagement.CountedResourcePool.tryAcquire(CountedResourcePool.java:51)

                at com.vontu.discover.crawler.framework.AsynchronousItemHandler.acquireMemoryResource(AsynchronousItemHandler.java:201)

                at com.vontu.discover.crawler.framework.AsynchronousItemHandler.ensureResourceAvailable(AsynchronousItemHandler.java:188)

                at com.vontu.discover.crawler.framework.AsynchronousItemHandler.createAndSubmitTask(AsynchronousItemHandler.java:151)

                at com.vontu.discover.crawler.framework.AsynchronousItemHandler.submit(AsynchronousItemHandler.java:106)

                at com.vontu.discover.crawler.framework.RepositoryCrawler.handleNextItem(RepositoryCrawler.java:730)

                at com.vontu.discover.crawler.framework.RepositoryCrawler.crawlContentRoot(RepositoryCrawler.java:636)

                at com.vontu.discover.crawler.framework.RepositoryCrawler.crawlUserContentRoot(RepositoryCrawler.java:428)

                at com.vontu.discover.crawler.framework.RepositoryCrawler.crawl(RepositoryCrawler.java:309)

                at com.vontu.discover.crawler.framework.RepositoryCrawler$CrawlerThread.run(RepositoryCrawler.java:215)

Dec 12, 2016 7:01:07 PM com.vontu.discover.crawler.framework.ContentItemProcessor$ItemFailedEventNotifier notifyObservers

WARNING: Failed to fetch item //Example.doc, cause: Failed to download.

2). 

Dec 13, 2016 12:01:38 AM com.vontu.filesystemcrawler.LastAccessDateResetterImpl resetDate

WARNING: Transport1 timedout waiting for response to SmbComSessionSetupAndX[command=SMB_COM_SESSION_SETUP_ANDX,received=false,errorCode=0,flags=0x0018,flags2=0xC803,signSeq=0,tid=0,pid=54874,uid=16387,mid=23702,wordCount=12,byteCount=239,andxCommand=0xFF,andxOffset=0,snd_buf_size=16644,maxMpxCount=10,VC_NUMBER=1,sessionKey=0,lmHash.length=0,ntHash.length=0,capabilities=-2147483564,accountName=null,primaryDomain=null,NATIVE_OS=Windows Server 2012 R2,NATIVE_LANMAN=jCIFS]

jcifs.smb.SmbException: Transport1 timedout waiting for response to SmbComSessionSetupAndX[command=SMB_COM_SESSION_SETUP_ANDX,received=false,errorCode=0,flags=0x0018,flags2=0xC803,signSeq=0,tid=0,pid=54874,uid=16387,mid=23702,wordCount=12,byteCount=239,andxCommand=0xFF,andxOffset=0,snd_buf_size=16644,maxMpxCount=10,VC_NUMBER=1,sessionKey=0,lmHash.length=0,ntHash.length=0,capabilities=-2147483564,accountName=null,primaryDomain=null,NATIVE_OS=Windows Server 2012 R2,NATIVE_LANMAN=jCIFS]

jcifs.util.transport.TransportException: Transport1 timedout waiting for response to SmbComSessionSetupAndX[command=SMB_COM_SESSION_SETUP_ANDX,received=false,errorCode=0,flags=0x0018,flags2=0xC803,signSeq=0,tid=0,pid=54874,uid=16387,mid=23702,wordCount=12,byteCount=239,andxCommand=0xFF,andxOffset=0,snd_buf_size=16644,maxMpxCount=10,VC_NUMBER=1,sessionKey=0,lmHash.length=0,ntHash.length=0,capabilities=-2147483564,accountName=null,primaryDomain=null,NATIVE_OS=Windows Server 2012 R2,NATIVE_LANMAN=jCIFS]

                at jcifs.util.transport.Transport.sendrecv(Transport.java:73)

                at jcifs.smb.SmbTransport.send(SmbTransport.java:655)

                at jcifs.smb.SmbSession.sessionSetup(SmbSession.java:390)

                at jcifs.smb.SmbSession.send(SmbSession.java:218)

                at jcifs.smb.SmbTree.treeConnect(SmbTree.java:176)

                at jcifs.smb.SmbFile.doConnect(SmbFile.java:914)

                at jcifs.smb.SmbFile.connect(SmbFile.java:960)

                at jcifs.smb.SmbFile.connect0(SmbFile.java:883)

                at jcifs.smb.SmbFile.queryPath(SmbFile.java:1377)

                at jcifs.smb.SmbFile.exists(SmbFile.java:1460)

                at jcifs.smb.SmbFile.setPathInformation(SmbFile.java:2654)

                at jcifs.smb.SmbFile.setAccessTime(SmbFile.java:2699)

                at com.vontu.filesystemcrawler.file.SmbFileAdapter.setLastAccessDate(SmbFileAdapter.java:179)

                at com.vontu.filesystemcrawler.LastAccessDateResetterImpl.resetDate(LastAccessDateResetterImpl.java:34)

                at com.vontu.filesystemcrawler.ContentFetcher.fetch(ContentFetcher.java:45)

                at com.vontu.filesystemcrawler.FileItem.fetch(FileItem.java:134)

                at com.vontu.filesystemcrawler.FileSystemContentItemProcessor.fetchItemContentAndMetaData(FileSystemContentItemProcessor.java:32)

                at com.vontu.discover.crawler.framework.ContentItemProcessor.process(ContentItemProcessor.java:31)

                at com.vontu.discover.crawler.framework.ItemProcessorTask.call(ItemProcessorTask.java:52)

                at com.vontu.discover.crawler.framework.ItemProcessorTask.call(ItemProcessorTask.java:10)

                at java.util.concurrent.FutureTask.run(FutureTask.java:266)

                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

                at java.lang.Thread.run(Thread.java:745)

 

                at jcifs.smb.SmbTransport.send(SmbTransport.java:660)

                at jcifs.smb.SmbSession.sessionSetup(SmbSession.java:390)

                at jcifs.smb.SmbSession.send(SmbSession.java:218)

                at jcifs.smb.SmbTree.treeConnect(SmbTree.java:176)

                at jcifs.smb.SmbFile.doConnect(SmbFile.java:914)

                at jcifs.smb.SmbFile.connect(SmbFile.java:960)

                at jcifs.smb.SmbFile.connect0(SmbFile.java:883)

                at jcifs.smb.SmbFile.queryPath(SmbFile.java:1377)

                at jcifs.smb.SmbFile.exists(SmbFile.java:1460)

                at jcifs.smb.SmbFile.setPathInformation(SmbFile.java:2654)

                at jcifs.smb.SmbFile.setAccessTime(SmbFile.java:2699)

                at com.vontu.filesystemcrawler.file.SmbFileAdapter.setLastAccessDate(SmbFileAdapter.java:179)

                at com.vontu.filesystemcrawler.LastAccessDateResetterImpl.resetDate(LastAccessDateResetterImpl.java:34)

                at com.vontu.filesystemcrawler.ContentFetcher.fetch(ContentFetcher.java:45)

                at com.vontu.filesystemcrawler.FileItem.fetch(FileItem.java:134)

                at com.vontu.filesystemcrawler.FileSystemContentItemProcessor.fetchItemContentAndMetaData(FileSystemContentItemProcessor.java:32)

                at com.vontu.discover.crawler.framework.ContentItemProcessor.process(ContentItemProcessor.java:31)

                at com.vontu.discover.crawler.framework.ItemProcessorTask.call(ItemProcessorTask.java:52)

                at com.vontu.discover.crawler.framework.ItemProcessorTask.call(ItemProcessorTask.java:10)

                at java.util.concurrent.FutureTask.run(FutureTask.java:266)

                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

                at java.lang.Thread.run(Thread.java:745)

Dec 13, 2016 12:01:38 AM com.vontu.discover.crawler.framework.ContentItemProcessor$ItemFailedEventNotifier notifyObservers

WARNING: Failed to fetch item //Example.xlsx, cause: Failed to download.

 

and finally 3).

Dec 13, 2016 12:01:39 AM com.vontu.filesystemcrawler.LastAccessDateResetterFactoryImpl getLastAccessDate

WARNING: The network name cannot be found.

jcifs.smb.SmbException: The network name cannot be found.

                at jcifs.smb.SmbTransport.checkStatus(SmbTransport.java:563)

                at jcifs.smb.SmbTransport.send(SmbTransport.java:663)

                at jcifs.smb.SmbSession.send(SmbSession.java:238)

                at jcifs.smb.SmbTree.treeConnect(SmbTree.java:176)

                at jcifs.smb.SmbFile.doConnect(SmbFile.java:914)

                at jcifs.smb.SmbFile.connect(SmbFile.java:960)

                at jcifs.smb.SmbFile.connect0(SmbFile.java:883)

                at jcifs.smb.SmbFile.queryPath(SmbFile.java:1377)

                at jcifs.smb.SmbFile.length(SmbFile.java:2517)

                at com.vontu.filesystemcrawler.file.SmbFileAdapter.getLength(SmbFileAdapter.java:109)

                at com.vontu.filesystemcrawler.LastAccessDateResetterFactoryImpl.getLastAccessDate(LastAccessDateResetterFactoryImpl.java:43)

                at com.vontu.filesystemcrawler.LastAccessDateResetterFactoryImpl.createResetter(LastAccessDateResetterFactoryImpl.java:30)

                at com.vontu.filesystemcrawler.ContentFetcher.fetch(ContentFetcher.java:38)

                at com.vontu.filesystemcrawler.FileItem.fetch(FileItem.java:134)

                at com.vontu.filesystemcrawler.FileSystemContentItemProcessor.fetchItemContentAndMetaData(FileSystemContentItemProcessor.java:32)

                at com.vontu.discover.crawler.framework.ContentItemProcessor.process(ContentItemProcessor.java:31)

                at com.vontu.discover.crawler.framework.ItemProcessorTask.call(ItemProcessorTask.java:52)

                at com.vontu.discover.crawler.framework.ItemProcessorTask.call(ItemProcessorTask.java:10)

                at java.util.concurrent.FutureTask.run(FutureTask.java:266)

                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

                at java.lang.Thread.run(Thread.java:745)

 

We're going to increase the initial java heap size on the detection servers as a start from:

 

# Initial Java Heap Size (in MB)

wrapper.java.initmemory = 4096

wrapper.java.maxmemory = 8192

to

# Initial Java Heap Size (in MB)

wrapper.java.initmemory = 6144

wrapper.java.maxmemory = 12288

 

 

 

I'd be very surprised if increasing the heap size did anything, there any reason you chose that particular setting specifically? Typically, you'd only increase the File Reader memory and only if you're using a large amount of cores/message chains and/or large indexes for EDM/IDM.

Did you check to see if the file share can be mapped via Windows on the Network Discover box?

Good Morning Dean,

 

This was the recommendation after the above logs were analyzed by our managed services provider. Unfortunetly, I'm rather new to my company and the environment and do not yet have access to the servers themselves, just the management GUI. Again, our scan accounts have domain admin access (not saying that some shares aren't locked down more than that) put that shouldn't be causing scans to repeated fail, right? My thought was that if the discover server couldn't either map to a drive or didn't have access, it would report the error in move on, not fail.

 

And just to be clear, these are our initial full scans of our entire DFS environment. A lot of these shares are multiple terabytes of data. Also, just found out yesterday that all of our servers are running dynamic memory allocation, not sure how that's effecting the scans.

Try a smaller target that you know the box has access to. If nothing else, target the Network Discover server itself and see if it gets further... such as \\<ip>\d$\ If the account has domain admin it should have access to the server if its on the same domain.