> ## Documentation index
> Fetch the complete documentation index at: https://developer.rhombus.com/llms.txt
> Use it to discover all available pages before exploring further.

# WebSocket Troubleshooting

> Diagnose and resolve common Rhombus WebSocket issues — connection failures, STOMP errors, missed heartbeats, authentication problems, and dropped events.

This guide covers common issues when working with Rhombus WebSocket connections and how to resolve them.

## Connection issues

### Unable to connect (connection refused)

**Symptoms**: WebSocket handshake fails immediately with a connection error.

**Possible Causes**:

- Firewall blocking outbound connections on port 8443
- Incorrect hostname

**Solutions**:

1. Verify outbound access to `ws.rhombussystems.com` on port `8443`:
   ```bash
   # Test connectivity
   curl -v https://ws.rhombussystems.com:8443/websocket

   # Test TCP connectivity directly
   nc -zv ws.rhombussystems.com 8443
   ```
2. Check that your network/firewall allows outbound WSS on port 8443
3. If behind a corporate proxy, ensure WebSocket upgrade requests are permitted

### HTTP 401 Unauthorized

**Symptoms**: WebSocket handshake fails with HTTP 401. The JSON body's `errorCode` says why.

**Possible Causes**:

- `UNAUTHENTICATED`: no credential reached the server. `x-auth-scheme` is missing or misspelled, the API key or OAuth token was sent as a query parameter (only a federated token may be), or the scheme is a certificate scheme (`api`, `partner-api`), which this connection doesn't accept
- `INVALID_CREDENTIAL`: the key or token is malformed, unknown (a deleted API key, for example), or from the other region
- `CREDENTIAL_EXPIRED` or `CREDENTIAL_REVOKED`: the key or token expired, or it was revoked (an OAuth token whose grant was revoked, for example). A federated token also stops working when the key, OAuth grant or session that minted it is deleted or revoked
- `CERTIFICATE_MISMATCH`: a certificate key sent with `api-token`

**Solutions**:

1. Send the scheme and the credential the way [WebSocket authentication](https://developer.rhombus.com/websocket/authentication#authentication-parameters) shows: `x-auth-scheme=api-token` with the key in the `x-auth-apikey` header, `Authorization: Bearer <token>` for an OAuth access token, or `x-auth-scheme=federated-token&x-auth-ft=<token>` for a federated token
2. Verify the key works with the REST API in the same region:
   ```bash
   curl -X POST https://api2.rhombussystems.com/api/org/getOrgV2 \
     -H "x-auth-scheme: api-token" \
     -H "x-auth-apikey: YOUR_TOKEN" \
     -H "Content-Type: application/json" \
     -d '{}'
   ```
3. Replace an expired or revoked credential: create a new key, refresh the OAuth token, or mint a new federated token

### HTTP 403 Forbidden

**Symptoms**: WebSocket handshake fails with HTTP 403. The credential is valid but isn't allowed on this connection; the JSON body's `errorCode` says why.

**Possible Causes**:

- `PERMISSION_REQUIRED`: a partner credential without the `x-auth-org` header, or an OAuth access token whose grant doesn't include the `read` scope (the response also carries `WWW-Authenticate: Bearer error="insufficient_scope", scope="read"`)
- `PARTNER_CLIENT_ACCESS_DENIED`: a partner credential not authorized for the organization in `x-auth-org`
- `DOMAIN_NOT_ALLOWED`: a federated token minted with `domain`, opened from a page whose `Origin` is a different origin

**Solutions**:

1. If using a partner credential, send the target organization's UUID in the `x-auth-org` **header** (the server reads `x-auth-org` from request headers, not the query string):

```text
   Header: x-auth-org: CLIENT_ORG_UUID
   URL:    wss://ws.rhombussystems.com:8443/websocket?x-auth-scheme=api-token
```

2. For an OAuth app, request the `read` scope when the user authorizes it
3. Mint federated tokens with your page's exact origin as `domain` (scheme, host and port), or without `domain`

### Certificate-based auth not working

**Symptoms**: Connection fails when using an mTLS certificate key: 401 `UNAUTHENTICATED` with the `api` scheme, or 401 `CERTIFICATE_MISMATCH` when the certificate key is sent with `api-token`.

**Cause**: The event WebSocket doesn't accept certificate keys.

**Solution**: Connect with a standard API key (Auth Type **Api Token**), an OAuth access token or a federated token. Certificate keys work on the REST API.

## STOMP protocol issues

### CONNECTED frame never received

**Symptoms**: WebSocket connects successfully, but no STOMP `CONNECTED` response after sending `CONNECT`.

**Possible Causes**:

- Malformed STOMP `CONNECT` frame
- Missing null terminator (`\x00`)
- Incorrect `accept-version` value

**Solutions**:

1. Verify your frame includes the null byte terminator:
   ```python
   # Correct
   frame = "CONNECT\naccept-version:1.2\nheart-beat:10000,10000\n\n\x00"

   # Wrong - missing \x00
   frame = "CONNECT\naccept-version:1.2\nheart-beat:10000,10000\n\n"
   ```
2. Ensure there is an empty line (`\n\n`) between headers and body
3. Use `accept-version:1.2` (not `1.0` or `1.1`)

### No messages received after subscribing

**Symptoms**: Connection and subscription succeed, but no MESSAGE frames arrive.

**Possible Causes**:

- The credential's role doesn't cover the devices or locations generating events. Events outside the role are dropped without an error, policy alerts included: an alert is delivered only to a role that can access its device or location.
- No events occurring in the organization
- Subscription frame malformed

A wrong organization UUID doesn't cause silence: subscribing to another organization's topic gets a STOMP `ERROR` frame, and the server closes the socket with code `1002`.

**Solutions**:

1. Check the role in the Rhombus Console (for an API key, the key's role; for an OAuth token, the user's role; for a federated token, the role of the credential that minted it): a full-access role receives every event in the organization. See [What a connection receives](https://developer.rhombus.com/websocket/authentication#what-a-connection-receives).
2. Verify your org UUID by calling the REST API:
   ```bash
   curl -X POST https://api2.rhombussystems.com/api/org/getOrgV2 \
     -H "x-auth-scheme: api-token" \
     -H "x-auth-apikey: YOUR_TOKEN" \
     -H "Content-Type: application/json" \
     -d '{}'
   ```
3. Trigger a test event (e.g., adjust a camera setting) to confirm the stream is active
4. Verify the topic format is exactly `/topic/change/{orgUuid}` with no trailing slashes or spaces

## Connection stability issues

### Connection drops every \~30 seconds

**Symptoms**: WebSocket connection closes after approximately 30 seconds of inactivity.

**Cause**: Heartbeats are not being sent.

**Solution**: Implement the heartbeat sender that sends `\n` every 10 seconds:

```python
import threading

def heartbeat_sender(ws, stop_event):
    while not stop_event.is_set():
        ws.send("\n")
        stop_event.wait(10)

stop = threading.Event()
thread = threading.Thread(target=heartbeat_sender, args=(ws, stop), daemon=True)
thread.start()
```

### Connection closed with code 1008

**Symptoms**: The server sends a STOMP `ERROR` frame and closes the connection with code `1008`.

**Cause**: The credential was deleted or revoked, or its role changed, while the connection was open. The server re-checks the credential about once a minute, so this happens within 60 seconds of the change. Revoking an OAuth grant, or the key or session that minted a federated token, has the same effect.

**Solution**: Check the credential before reconnecting. After a role change, a new connection picks up the new role; after a delete or revoke, every handshake with that credential fails with 401, so replace it instead of retrying.

### Connection drops intermittently

**Symptoms**: Connection works for a while then drops unpredictably.

**Possible Causes**:

- Network instability
- Server-side maintenance
- Load balancer timeout

**Solutions**:

1. Implement automatic reconnection with backoff:
   ```python
   while True:
       try:
           run_websocket_monitor()
       except ConnectionError:
           print("Reconnecting in 5 seconds...")
           time.sleep(5)
   ```
2. Log disconnection reasons to identify patterns
3. Monitor heartbeat receipt to detect dead connections proactively

### High memory usage over time

**Symptoms**: Application memory grows steadily while connected.

**Possible Causes**:

- Events accumulating in an unbounded queue
- Not processing messages fast enough

**Solutions**:

1. Use bounded message buffers:
   ```python
   # Use a bounded queue
   from queue import Queue
   event_queue = Queue(maxsize=1000)
   ```
2. Process events asynchronously to prevent backpressure
3. Drop or log events if the processing queue is full

## Event processing issues

### Missing events

**Symptoms**: Some events that appear in the Rhombus console don't appear in the WebSocket stream.

**Possible Causes**:

- The credential's role doesn't cover those devices or locations (events outside the role are dropped without an error)
- The role is limited and the entity type has no access check: those types go only to full-access roles
- Event filtering is too aggressive
- Brief disconnection caused missed events
- Event occurred before subscription was established

**Solutions**:

1. Temporarily enable all events to verify the stream:
   ```python
   # Don't filter by entity type
   if frame["command"] == "MESSAGE":
       print(json.loads(frame["body"]))
   ```
2. WebSocket events are real-time only. For historical events, use the REST API
3. Ensure SUBSCRIBE is sent before expecting events

### Malformed JSON in event body

**Symptoms**: `json.loads()` fails on MESSAGE body.

**Possible Causes**:

- Frame parser not correctly separating body from headers
- Null bytes or extra whitespace in the body

**Solutions**:

1. Strip null bytes before parsing:
   ```python
   body = frame["body"].strip("\x00").strip()
   payload = json.loads(body)
   ```
2. Verify your frame parser splits on `\n\n` correctly (first occurrence only)

## Debugging tools

### Enable raw frame logging

Add logging to see exactly what's being sent and received:

```python
import logging

logging.basicConfig(level=logging.DEBUG)

# In your message loop:
raw = ws.recv()
logging.debug(f"Received: {repr(raw)}")
```

### Test with websocat

Use `wscat` to manually interact with the WebSocket endpoint:

```bash
# Install
npm install -g wscat

# Connect (note: wscat does not support custom headers natively,
# so use a tool like websocat for full header support)
websocat -H "x-auth-apikey: YOUR_TOKEN" \
  "wss://ws.rhombussystems.com:8443/websocket?x-auth-scheme=api-token"

# Then manually type STOMP frames:
CONNECT
accept-version:1.2
heart-beat:10000,10000

^@
```

> **Note**: `^@` represents the null byte (`\x00`). In most terminals, type `Ctrl+@` or `Ctrl+Shift+2` to produce it.

### Rhombus CLI monitor

Use the [Rhombus CLI](https://developer.rhombus.com/rhombus-cli) to verify your account and WebSocket endpoint work:

```bash
# Install the CLI (macOS/Linux)
brew install RhombusSystems/tap/rhombus

# Authenticate
rhombus login

# Test WebSocket monitoring
rhombus monitor --all-events --json
```

If the CLI works but your code doesn't, compare the connection parameters your code uses against the CLI's behavior. See the [Rhombus CLI documentation](https://developer.rhombus.com/rhombus-cli) for full installation and usage details.

## Getting help

If you've tried the steps above and are still experiencing issues:

1. Check the [changelog](https://developer.rhombus.com/changelog) for recent changes
2. Contact Rhombus Support at <support@rhombus.com> with:
   - Your organization UUID
   - The error message or HTTP status code
   - The timestamp of the failed connection attempt
   - Your client language and WebSocket library version