Cloud-Delivered Web Security Services

Hi Partners,

Good day to all.

May I know if is there any of you used the SyncAPI for WSS to download access from the WSS? I understand that the scripting portion is not covered by Symantec and is not diuscussed on the guide from the link below.

https://symwisedownload.symantec.com//resources/sites/SYMWISE/content/live/DOCUMENTATION/10000/DOC10379/en_US/PDF_SyncAPI.pdf?__gda__=1517389182_ed21d95f7d0d298ecb84a51b3a24ea83

is there anyone able to make it work? If so, possible to share some ideas.

I was actually provided a sample script (cloud-log-downloader.sh), but I am stucked with the error below.

--------------------------------------------------------------------------------------------------------------------------------

$ sh cloud-log-downloader.sh
Tue, Jan 30, 2018 12:48:06 AM
No arguments provided!

Syntax: cloud-log-downloader.sh [OPTIONS] /path/to/configuration/file
Available [OPTIONS]:
-d : Debug mode.
-last-month : Download the last month of logs (use only manually as an initial download).
-last-week : Download the last week of logs (use only manually as an initial download).
-last-day : Download the last day of logs (use only manually as an initial download).

--------------------------------------------------------------------------------------------------------------------------------

Hope to hear any response from anyone.

Thanks and regards,

JBM

Hi JBM,

no scripts here - but I got it to work using curl and an ubuntu box. The only thing I'm struggling with is the time ranges, as they do not seem to compile for me (always get a 400 Bad Request with Invalid start or end date). However, using 0 values, the download seems to kick off nicely:

curl -H "X-APIUsername: SomeUser" -H "X-APIPassword: SomePassword" --data-binary --url "https://portal.threatpulse.com/reportpod/logs/sync?startDate=0&endDate=0&token=none" -o cloudarchive.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 35.1M    0 35.1M    0     5  5063k      0 --:--:--  0:00:07 --:--:-- 6701k

Hope it helps,

OK found the Timerange issue.

Timestamps should be on the hour for the logs to accumulate, but they need to be specified in milliseconds:

https://portal.threatpulse.com/reportpod/logs/sync?startDate=1521158400000&endDate=1521205200000&token=none

This results in:

root@boxnuc5i3ryh:/var/log/WSS# curl -H "X-APIUsername: SomeUser" -H "X-APIPassword: SomePAssword" --data-binary --url "https://portal.threatpulse.com/reportpod/logs/sync?startDate=1521158400000&endDate=1521205200000&token=none" -o CombinedLogs_201816030000.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 11.5M    0 11.5M  100     5  2536k      1  0:00:05  0:00:04  0:00:01 2671k

root@boxnuc5i3ryh:/var/log/WSS# ls -la
total 23740
drwxr-xr-x  2 root root       4096 Mar 16 20:51 .
drwxrwxr-x 15 root syslog     4096 Mar 16 20:35 ..
-r--r-----  1 root root     272263 Mar 16 19:47 cloud_XXXXX_20180316000000.log.gz
-r--r-----  1 root root     284400 Mar 16 19:47 cloud_XXXXX_20180316010000.log.gz
-r--r-----  1 root root     285731 Mar 16 19:47 cloud_XXXXX_20180316020000.log.gz
-r--r-----  1 root root     298974 Mar 16 19:47 cloud_XXXXX_20180316030000.log.gz
-r--r-----  1 root root     285197 Mar 16 19:47 cloud_XXXXX_20180316040000.log.gz
-r--r-----  1 root root     280789 Mar 16 19:47 cloud_XXXXX_20180316050000.log.gz
-r--r-----  1 root root     824923 Mar 16 19:47 cloud_XXXXX_20180316060000.log.gz
-r--r-----  1 root root    1423861 Mar 16 19:47 cloud_XXXXX_20180316070000.log.gz
-r--r-----  1 root root    1363443 Mar 16 19:47 cloud_XXXXX_20180316080000.log.gz
-r--r-----  1 root root    1532602 Mar 16 19:47 cloud_XXXXX_20180316090000.log.gz
-r--r-----  1 root root    1274500 Mar 16 19:47 cloud_XXXXX_20180316100000.log.gz
-r--r-----  1 root root    1719019 Mar 16 19:47 cloud_XXXXX_20180316110000.log.gz
-r--r-----  1 root root    1123836 Mar 16 19:47 cloud_XXXXX_20180316120000.log.gz
-r--r-----  1 root root    1166654 Mar 16 19:47 cloud_XXXXX_20180316130000.log.gz
root@boxnuc5i3ryh:/var/log/WSS# gunzip *.gz
root@boxnuc5i3ryh:/var/log/WSS# ls -la
total 68244
drwxr-xr-x  2 root root       4096 Mar 16 20:51 .
drwxrwxr-x 15 root syslog     4096 Mar 16 20:35 ..
-r--r-----  1 root root     999594 Mar 16 19:47 cloud_XXXXX_20180316000000.log
-r--r-----  1 root root    1009841 Mar 16 19:47 cloud_XXXXX_20180316010000.log
-r--r-----  1 root root    1031117 Mar 16 19:47 cloud_XXXXX_20180316020000.log
-r--r-----  1 root root    1099513 Mar 16 19:47 cloud_XXXXX_20180316030000.log
-r--r-----  1 root root    1011948 Mar 16 19:47 cloud_XXXXX_20180316040000.log
-r--r-----  1 root root    1026054 Mar 16 19:47 cloud_XXXXX_20180316050000.log
-r--r-----  1 root root    5115317 Mar 16 19:47 cloud_XXXXX_20180316060000.log
-r--r-----  1 root root    8889669 Mar 16 19:47 cloud_XXXXX_20180316070000.log
-r--r-----  1 root root    8588719 Mar 16 19:47 cloud_XXXXX_20180316080000.log
-r--r-----  1 root root    9848640 Mar 16 19:47 cloud_XXXXX_20180316090000.log
-r--r-----  1 root root    7046214 Mar 16 19:47 cloud_XXXXX_20180316100000.log
-r--r-----  1 root root   10357411 Mar 16 19:47 cloud_XXXXX_20180316110000.log
-r--r-----  1 root root    6729116 Mar 16 19:47 cloud_XXXXX_20180316120000.log
-r--r-----  1 root root    7093665 Mar 16 19:47 cloud_XXXXX_20180316130000.log
root@boxnuc5i3ryh:/var/log/WSS# 

Andy, for those of us who aren't bash scripting savvy, can you provide the method you used to determine the timestamp to start from and end on (or scripts and/or cronjobs) for generating the timestamp as required?

Regards,

Lucas

Hi Lucas,

try something like this:

vi /var/tmp/myscript.sh

copy-paste the below
 

#/bin/bash

logstart=$(date -d "`date -d yesterday +%F`" +%s000)
logend=$(date -d "`date -d today +%F`" +%s000)
datestamp=$(date -d yesterday +%Y%m%d)

curl -H "X-APIUsername: SomeUser" -H "X-APIPassword: SomePassword" --data-binary --url "https://portal.threatpulse.com/reportpod/logs/sync?startDate=$logstart&endDate=$logend&token=none" -o /var/logs/WSS/CombinedLogs_$datestamp.zip

exit 0

and replace the obvious SomeUser, SomePassword with your API credentials, and make sure the output dir (/var/logs/WSS) is changed in the script or created on your system, and to run this every night:

crontab -e

0 0 * * * /var/tmp/myscript.sh

Andy,

Thanks for the tips!

In parallel, I made progress getting it working in PowerShell, which was easier than I thought. Here's my code:

# things to improve:
# - keep track of the last successful download or figure out how to grab the token
# - Change console output to log output
# - Unzip the files and move them where they need to be

$str_XAPIUsername = "SomeAPIUser"
$str_XAPIPassword = "SomePassword"
$outfileName = "d:\path\to\cloud_logs.zip"

# I want to get everything up until now
$currTime = (get-date)
$endDate = 0

# get the current hour with zero minutes, seconds, and milliseconds. 
# This is important because the API seems to want to give out hourly updates
$hourStart = get-date -minute 0 -second 0 -mill 0
$startDate = ([DateTimeOffset]$hourStart).ToUnixTimeMilliseconds()

write-host "Will collect logs from $hourStart THROUGH $currTime (enddate = $enddate)" -foreground black -background green
write-host "File will be written to $output" -foreground black -background green
$uri_full = "https://portal.threatpulse.com/reportpod/logs/sync?&startDate="+$startdate+"&endDate="+$endDate+"&token=none"
write-host $uri_full -background green -foreground black
# Note: must use -output $filename and not | out-file 
invoke-restmethod -verbose -headers @{"X-APIUsername" = $str_XAPIUsername; "X-APIPassword" = $str_XAPIPassword; "method" = $method;} -uri $uri_full -outfile $output

Thanks for that as well - no powershell skills here so this will certainly help when a customer asks this. I've actually let the scripting slide, as we use the TA developed for splunk which is available on the support site. That one works excellent!

Glad to be of assistance. Once I get a working script, do you know if there is a place to submit the script for the community, or do you think just leaving it in the thread will suffice?

Unfortunately, not everyone uses Splunk (ca-ching), so having a generic way to grab the logs and then hand them off to our syslog collector is important.

Hi Lucas, maybe if you create the content here (https://www.symantec.com/connect/node/add/article) and link to it from this thread - people will find it when searching the Symantec Connect Community. I'll see if I can create a decent script in Perl or Python to make it cross-platform usable.