WebSocket Troubleshooting

This guide covers common issues when working with Rhombus WebSocket connections and how to resolve them.

Connection issues

Unable to connect (connection refused)

Symptoms: WebSocket handshake fails immediately with a connection error.

Possible Causes:

• Firewall blocking outbound connections on port 8443
• Incorrect hostname

Solutions:

1. Verify outbound access to ws.rhombussystems.com on port 8443:
   bash

   # Test connectivity
   curl -v https://ws.rhombussystems.com:8443/websocket
   
   # Test TCP connectivity directly
   nc -zv ws.rhombussystems.com 8443

2. Check that your network/firewall allows outbound WSS on port 8443
3. If behind a corporate proxy, ensure WebSocket upgrade requests are permitted

HTTP 401 Unauthorized

Symptoms: WebSocket handshake fails with HTTP 401. The JSON body's errorCode says why.

Possible Causes:

• UNAUTHENTICATED: no credential reached the server. x-auth-scheme is missing or misspelled, the API key or OAuth token was sent as a query parameter (only a federated token may be), or the scheme is a certificate scheme (api, partner-api), which this connection doesn't accept
• INVALID_CREDENTIAL: the key or token is malformed, unknown (a deleted API key, for example), or from the other region
• CREDENTIAL_EXPIRED or CREDENTIAL_REVOKED: the key or token expired, or it was revoked (an OAuth token whose grant was revoked, for example). A federated token also stops working when the key, OAuth grant or session that minted it is deleted or revoked
• CERTIFICATE_MISMATCH: a certificate key sent with api-token

Solutions:

1. Send the scheme and the credential the way WebSocket authentication shows: x-auth-scheme=api-token with the key in the x-auth-apikey header, Authorization: Bearer <token> for an OAuth access token, or x-auth-scheme=federated-token&x-auth-ft=<token> for a federated token
2. Verify the key works with the REST API in the same region:
   bash

   curl -X POST https://api2.rhombussystems.com/api/org/getOrgV2 \
     -H "x-auth-scheme: api-token" \
     -H "x-auth-apikey: YOUR_TOKEN" \
     -H "Content-Type: application/json" \
     -d '{}'

3. Replace an expired or revoked credential: create a new key, refresh the OAuth token, or mint a new federated token

HTTP 403 Forbidden

Symptoms: WebSocket handshake fails with HTTP 403. The credential is valid but isn't allowed on this connection; the JSON body's errorCode says why.

Possible Causes:

• PERMISSION_REQUIRED: a partner credential without the x-auth-org header, or an OAuth access token whose grant doesn't include the read scope (the response also carries WWW-Authenticate: Bearer error="insufficient_scope", scope="read")
• PARTNER_CLIENT_ACCESS_DENIED: a partner credential not authorized for the organization in x-auth-org
• DOMAIN_NOT_ALLOWED: a federated token minted with domain, opened from a page whose Origin is a different origin

Solutions:

1. If using a partner credential, send the target organization's UUID in the x-auth-org header (the server reads x-auth-org from request headers, not the query string):

   Header: x-auth-org: CLIENT_ORG_UUID
   URL:    wss://ws.rhombussystems.com:8443/websocket?x-auth-scheme=api-token

2. For an OAuth app, request the read scope when the user authorizes it
3. Mint federated tokens with your page's exact origin as domain (scheme, host and port), or without domain

Certificate-based auth not working

Symptoms: Connection fails when using an mTLS certificate key: 401 UNAUTHENTICATED with the api scheme, or 401 CERTIFICATE_MISMATCH when the certificate key is sent with api-token.

Cause: The event WebSocket doesn't accept certificate keys.

Solution: Connect with a standard API key (Auth Type Api Token), an OAuth access token or a federated token. Certificate keys work on the REST API.

STOMP protocol issues

CONNECTED frame never received

Symptoms: WebSocket connects successfully, but no STOMP CONNECTED response after sending CONNECT.

Possible Causes:

• Malformed STOMP CONNECT frame
• Missing null terminator (\x00)
• Incorrect accept-version value

Solutions:

1. Verify your frame includes the null byte terminator:
   python

   # Correct
   frame = "CONNECT\naccept-version:1.2\nheart-beat:10000,10000\n\n\x00"
   
   # Wrong - missing \x00
   frame = "CONNECT\naccept-version:1.2\nheart-beat:10000,10000\n\n"

2. Ensure there is an empty line (\n\n) between headers and body
3. Use accept-version:1.2 (not 1.0 or 1.1)

No messages received after subscribing

Symptoms: Connection and subscription succeed, but no MESSAGE frames arrive.

Possible Causes:

• The credential's role doesn't cover the devices or locations generating events. Events outside the role are dropped without an error, policy alerts included: an alert is delivered only to a role that can access its device or location.
• No events occurring in the organization
• Subscription frame malformed

A wrong organization UUID doesn't cause silence: subscribing to another organization's topic gets a STOMP ERROR frame, and the server closes the socket with code 1002.

Solutions:

1. Check the role in the Rhombus Console (for an API key, the key's role; for an OAuth token, the user's role; for a federated token, the role of the credential that minted it): a full-access role receives every event in the organization. See What a connection receives.
2. Verify your org UUID by calling the REST API:
   bash

   curl -X POST https://api2.rhombussystems.com/api/org/getOrgV2 \
     -H "x-auth-scheme: api-token" \
     -H "x-auth-apikey: YOUR_TOKEN" \
     -H "Content-Type: application/json" \
     -d '{}'

3. Trigger a test event (e.g., adjust a camera setting) to confirm the stream is active
4. Verify the topic format is exactly /topic/change/{orgUuid} with no trailing slashes or spaces

Connection stability issues

Connection drops every ~30 seconds

Symptoms: WebSocket connection closes after approximately 30 seconds of inactivity.

Cause: Heartbeats are not being sent.

Solution: Implement the heartbeat sender that sends \n every 10 seconds:

import threading

def heartbeat_sender(ws, stop_event):
    while not stop_event.is_set():
        ws.send("\n")
        stop_event.wait(10)

stop = threading.Event()
thread = threading.Thread(target=heartbeat_sender, args=(ws, stop), daemon=True)
thread.start()

Connection closed with code 1008

Symptoms: The server sends a STOMP ERROR frame and closes the connection with code 1008.

Cause: The credential was deleted or revoked, or its role changed, while the connection was open. The server re-checks the credential about once a minute, so this happens within 60 seconds of the change. Revoking an OAuth grant, or the key or session that minted a federated token, has the same effect.

Solution: Check the credential before reconnecting. After a role change, a new connection picks up the new role; after a delete or revoke, every handshake with that credential fails with 401, so replace it instead of retrying.

Connection drops intermittently

Symptoms: Connection works for a while then drops unpredictably.

Possible Causes:

• Network instability
• Server-side maintenance
• Load balancer timeout

Solutions:

1. Implement automatic reconnection with backoff:
   python

   while True:
       try:
           run_websocket_monitor()
       except ConnectionError:
           print("Reconnecting in 5 seconds...")
           time.sleep(5)

2. Log disconnection reasons to identify patterns
3. Monitor heartbeat receipt to detect dead connections proactively

High memory usage over time

Symptoms: Application memory grows steadily while connected.

Possible Causes:

• Events accumulating in an unbounded queue
• Not processing messages fast enough

Solutions:

1. Use bounded message buffers:
   python

   # Use a bounded queue
   from queue import Queue
   event_queue = Queue(maxsize=1000)

2. Process events asynchronously to prevent backpressure
3. Drop or log events if the processing queue is full

Event processing issues

Missing events

Symptoms: Some events that appear in the Rhombus console don't appear in the WebSocket stream.

Possible Causes:

• The credential's role doesn't cover those devices or locations (events outside the role are dropped without an error)
• The role is limited and the entity type has no access check: those types go only to full-access roles
• Event filtering is too aggressive
• Brief disconnection caused missed events
• Event occurred before subscription was established

Solutions:

1. Temporarily enable all events to verify the stream:
   python

   # Don't filter by entity type
   if frame["command"] == "MESSAGE":
       print(json.loads(frame["body"]))

2. WebSocket events are real-time only. For historical events, use the REST API
3. Ensure SUBSCRIBE is sent before expecting events

Malformed JSON in event body

Symptoms: json.loads() fails on MESSAGE body.

Possible Causes:

• Frame parser not correctly separating body from headers
• Null bytes or extra whitespace in the body

Solutions:

1. Strip null bytes before parsing:
   python

   body = frame["body"].strip("\x00").strip()
   payload = json.loads(body)

2. Verify your frame parser splits on \n\n correctly (first occurrence only)

Debugging tools

Enable raw frame logging

Add logging to see exactly what's being sent and received:

import logging

logging.basicConfig(level=logging.DEBUG)

# In your message loop:
raw = ws.recv()
logging.debug(f"Received: {repr(raw)}")

Test with websocat

Use wscat to manually interact with the WebSocket endpoint:

# Install
npm install -g wscat

# Connect (note: wscat does not support custom headers natively,
# so use a tool like websocat for full header support)
websocat -H "x-auth-apikey: YOUR_TOKEN" \
  "wss://ws.rhombussystems.com:8443/websocket?x-auth-scheme=api-token"

# Then manually type STOMP frames:
CONNECT
accept-version:1.2
heart-beat:10000,10000

^@

Rhombus CLI monitor

Use the Rhombus CLI to verify your account and WebSocket endpoint work:

# Install the CLI (macOS/Linux)
brew install RhombusSystems/tap/rhombus

# Authenticate
rhombus login

# Test WebSocket monitoring
rhombus monitor --all-events --json

If the CLI works but your code doesn't, compare the connection parameters your code uses against the CLI's behavior. See the Rhombus CLI documentation for full installation and usage details.

Getting help

If you've tried the steps above and are still experiencing issues:

1. Check the changelog for recent changes
2. Contact Rhombus Support at support@rhombus.com with:
   • Your organization UUID
   • The error message or HTTP status code
   • The timestamp of the failed connection attempt
   • Your client language and WebSocket library version