I have finally resolved the issue.
Never heard about having too much RAM could get you into trouble.......
Stumbled over the following kb article:
http://support.microsoft.com/kb/2634907/en-us
-----
TCP/IP packets that are received out of sequence are discarded in Windows 7 or in Windows Server 2008 R2
Assume that you install Windows 7 or Windows Server 2008 R2 Standard Edition on a computer that has more than 32 gigabytes (GB) of RAM. When the computer tries to communicate with another computer by using the TCP/IP protocol, all the TCP/IP packets that are received out of sequence are discarded. Additionally, the fragmented packets cannot be reassembled.
Note This issue causes retransmission overhead. Additionally, all network-related activity is affected. For example, when you try to perform a file transfer or perform a remote access session on the computer, network performance is affected by the retransmission overhead.
------
The solution was installing the hotfix provided by Microsoft in the kb.
The server had 36 GB RAM. But of cause only 32 GB can be utilized on Windows Server 2008 R2 Standard (SBS 2011)