Trying to connect to RDS - Sometimes takes 3 attempts for the connection to establish

I have RDS 2012 set up ( only 1 server configuration ) and when I and my clients attempt to connect to RDS using the remote desktop feed, sometimes it works on the first try and sometimes it works on the 3rd connection attempt.

When it doesn't work, I get the error :

"Remote Desktop can't connect to the remote PC for one of these reasons:

1)Your user account is not authorized to access the RD Gateway.

2)Your PC is not authorized to access the RD Gateway.

This error just seems like a generic error because I am authorized and it does work sometimes.

Any idea why this is happening? Let me know if I need to provide more info.

Thank you!

Patrick.

Also, might not be related but I'm seeing this sometimes in the event viewer,

Event code: 3005 
Event message: An unhandled exception has occurred. 
Event time: 5/19/2015 2:37:48 AM 
Event time (UTC): 5/19/2015 6:37:48 AM 
Event ID: 5c52a507bf6c4542947ae5e97b4290e1 
Event sequence: 12108 
Event occurrence: 300 
Event detail code: 0 
 
Application information: 
    Application domain: /LM/W3SVC/1/ROOT/RDWeb/Feed-3-130667854185997988 
    Trust level: Full 
    Application Virtual Path: /RDWeb/Feed 
    Application Path: C:\Windows\Web\RDWeb\Feed\ 
    Machine name: QU1CRPRDS01 
 
Process information: 
    Process ID: 3208 
    Process name: w3wp.exe 
    Account name: IIS APPPOOL\RDWebAccess 
 
Exception information: 
    Exception type: NullReferenceException 
    Exception message: Object reference not set to an instance of an object.
   at Microsoft.TerminalServices.Publishing.Portal.RWSCPUBAccessor.PrepareRdpFile(String& rdp_contents, String fileExtension, String ts_name, String alias, String& rdp_file_url, String& rdp_file_name, DateTime requestTime)
   at Microsoft.TerminalServices.Publishing.Portal.RWSCPUBAccessor.ProcessPublishedApp(String alias, String name, String resourceFile, String fileExtension, Byte[] iconContents, UInt32 pceIconSize, Int32 resourceType, FileExtensionInfo[] fileAssociations, String[] folderNames, String securityDescriptor, ArrayList& apps, DateTime requestTime, UInt64 lastUpdateTime)
   at Microsoft.TerminalServices.Publishing.Portal.RWSCPUBAccessor.GetFreshApps(DateTime requestTime)
   at Microsoft.TerminalServices.Publishing.Portal.RWSCPUBAccessor.OnCacheItemRemoved(String key, Object value, CacheItemRemovedReason removedReason)
   at System.Web.Caching.CacheEntry.CallCacheItemRemovedCallback(CacheItemRemovedCallback callback, CacheItemRemovedReason reason)

 
 
Request information: 
    Request URL:  
    Request path:  
    User host address:  
    User:  
    Is authenticated: False 
    Authentication Type:  
    Thread account name: IIS APPPOOL\RDWebAccess 
 
Thread information: 
    Thread ID: 60 
    Thread account name: IIS APPPOOL\RDWebAccess 
    Is impersonating: False 
    Stack trace:    at Microsoft.TerminalServices.Publishing.Portal.RWSCPUBAccessor.PrepareRdpFile(String& rdp_contents, String fileExtension, String ts_name, String alias, String& rdp_file_url, String& rdp_file_name, DateTime requestTime)
   at Microsoft.TerminalServices.Publishing.Portal.RWSCPUBAccessor.ProcessPublishedApp(String alias, String name, String resourceFile, String fileExtension, Byte[] iconContents, UInt32 pceIconSize, Int32 resourceType, FileExtensionInfo[] fileAssociations, String[] folderNames, String securityDescriptor, ArrayList& apps, DateTime requestTime, UInt64 lastUpdateTime)
   at Microsoft.TerminalServices.Publishing.Portal.RWSCPUBAccessor.GetFreshApps(DateTime requestTime)
   at Microsoft.TerminalServices.Publishing.Portal.RWSCPUBAccessor.OnCacheItemRemoved(String key, Object value, CacheItemRemovedReason removedReason)
   at System.Web.Caching.CacheEntry.CallCacheItemRemovedCallback(CacheItemRemovedCallback callback, CacheItemRemovedReason reason)
 
 
Custom event details: