Signal service questions

PhilippeVaucher · July 11, 2025, 8:03am

Describe the problem

When trying to run the signal service as standalone, netbird status reports it’s connected to the signal service but peers cannot ping each other and are stuck “connecting”.

To Reproduce

Run signal on signal.netbird.domain.com like this:

services:
  signal:
    image: 'netbirdio/signal:latest'
    command: "--log-level debug --letsencrypt-domain signal.netbird.domain.com --ssl-dir=/etc"
    ports:
      - '443:443'
      - '10000:10000'
    volumes:
      - './data/signal/letsencrypt:/etc/letsencrypt'
      - './data/signal/logs:/var/log/netbird'

Configure management.json like this

  "Signal": {
    "Proto": "https",
    "URI": "signal.netbird.domain.com:443",
    "Username": "",
    "Password": null
  },

Expected behavior

Peers connect as usual.

Are you using NetBird Cloud?

Self hosted.

NetBird version

0.48.0

Is any other VPN software installed?

No

Debug output

netbird status -d from a peer:

 cam1.netbird.selfhosted:
  NetBird IP: 100.70.205.198
  Public key: oWxONSNjVcitGblH6DA5OGDTh6PEIM6m4UwU37CODUE=
  Status: Connecting
  -- detail --
  Connection type:
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address:
  Last connection update: 1 minute, 45 seconds ago
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: false
  Networks: -
  Latency: 0s

 rut241.netbird.selfhosted:
  NetBird IP: 100.70.214.129
  Public key: 3JJDdMyn0VRXgaf7j7fVAgduInFMiuo5Vl69oe1MDh4=
  Status: Connecting
  -- detail --
  Connection type:
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address:
  Last connection update: 1 minute, 45 seconds ago
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: false
  Networks: -
  Latency: 0s

Events:
  [INFO] SYSTEM (04e29ca9-2eb8-4b25-8fee-7ba50191af4a)
    Message: Network map updated
    Time: 1 second ago
OS: windows/amd64
Daemon version: 0.46.0
CLI version: 0.46.0
Management: Connected to https://netbird.domain.com:443
Signal: Connected to https://signal.netbird.domain.com:443
Relays:
  [stun:netbird.domain.com:3478] is Available
  [turn:netbird.domain.com:3478?transport=udp] is Available
  [rels://de1.relay.netbird.domain.com:443] is Available
Nameservers:
  [8.8.8.8:53, 8.8.4.4:53] for [.] is Available
FQDN: stvs-pv-laptop1.netbird.selfhosted
NetBird IP: 100.70.63.243/16
Interface type: Userspace
Quantum resistance: false
Lazy connection: false
Networks: -
Forwarding rules: 0
Peers count: 1/34 Connected

The only peer connected is a LAN device.

Signal logs:

2025-06-25T13:30:41+02:00 INFO encryption/letsencrypt.go:22: running with LetsEncrypt ([signal.netbird.domain.com]). Cert will be stored in /etc/letsencrypt
2025-06-25T13:30:41+02:00 INFO signal/cmd/run.go:204: setting up TLS with LetsEncrypt.
2025-06-25T13:30:41+02:00 DEBG signal/cmd/run.go:179: Starting pprof server on 127.0.0.1:6060
2025-06-25T13:30:41+02:00 INFO signal/cmd/run.go:103: running metrics server: :9090/metrics
2025-06-25T13:30:41+02:00 INFO signal/cmd/run.go:230: running HTTP server (LetsEncrypt challenge handler) and gRPC server on the same port: [::]:443
2025-06-25T13:30:41+02:00 INFO signal/cmd/run.go:141: running gRPC backward compatibility server: [::]:10000
2025-06-25T13:30:41+02:00 INFO signal/cmd/run.go:144: signal server version 0.48.0
2025-06-25T13:30:41+02:00 INFO signal/cmd/run.go:145: started Signal Service
2025-06-25T13:31:37+02:00 DEBG signal/server/signal.go:108: registering new peer
2025-06-25T13:31:37+02:00 DEBG signal/peer/peer.go:88: peer registered [aIexxfEE7S8iTgtMEYU9375i5a+g119r3qqpBaRezkg=]
2025-06-25T13:31:37+02:00 DEBG signal/server/signal.go:100: peer connected [aIexxfEE7S8iTgtMEYU9375i5a+g119r3qqpBaRezkg=] [streamID 1750851097757985714]
2025-06-25T13:31:51+02:00 DEBG signal/server/signal.go:108: registering new peer
2025-06-25T13:31:51+02:00 DEBG signal/peer/peer.go:88: peer registered [BK1kwiMi55Vai1e1WhEtdal3sT2pueG5l1E+fCgurmg=]
2025-06-25T13:31:51+02:00 DEBG signal/server/signal.go:100: peer connected [BK1kwiMi55Vai1e1WhEtdal3sT2pueG5l1E+fCgurmg=] [streamID 1750851111088141618]
2025-06-25T13:33:34+02:00 DEBG signal/server/signal.go:103: peer stream closing [aIexxfEE7S8iTgtMEYU9375i5a+g119r3qqpBaRezkg=] [streamID 1750851097757985714]
2025-06-25T13:33:34+02:00 DEBG signal/server/signal.go:133: peer disconnected [aIexxfEE7S8iTgtMEYU9375i5a+g119r3qqpBaRezkg=] [streamID 1750851097757985714]
2025-06-25T13:33:34+02:00 DEBG signal/peer/peer.go:112: peer deregistered [aIexxfEE7S8iTgtMEYU9375i5a+g119r3qqpBaRezkg=]
2025-06-25T13:33:34+02:00 DEBG dispatcher@v0.0.0-20250514131221-a464fd5f30cb/dispatcher.go:69: stream cl osed for peer aIexxfEE7S8iTgtMEYU9375i5a+g119r3qqpBaRezkg=
2025-06-25T13:33:42+02:00 DEBG signal/server/signal.go:103: peer stream closing [BK1kwiMi55Vai1e1WhEtdal3sT2pueG5l1E+fCgurmg=] [streamID 1750851111088141618]
2025-06-25T13:33:42+02:00 DEBG signal/server/signal.go:133: peer disconnected [BK1kwiMi55Vai1e1WhEtdal3sT2pueG5l1E+fCgurmg=] [streamID 1750851111088141618]
2025-06-25T13:33:42+02:00 DEBG signal/peer/peer.go:112: peer deregistered [BK1kwiMi55Vai1e1WhEtdal3sT2pueG5l1E+fCgurmg=]

Screenshots

No screenshots.

Additional context

Actually I don’t see how to secure the signal service at all:

Flags:
  -h, --help                        help for run
      --letsencrypt-domain string   a domain to issue Let's Encrypt certificate for. Enables TLS using Let's Encrypt. Will fetch and renew certificate, and run the server with TLS
      --port int                    Server port to listen on (e.g. 10000) (default 10000)
      --ssl-dir string              server ssl directory location. *Required only for Let's Encrypt certificates. (default "/var/lib/netbird/")
      --cert-file string            Location of your SSL certificate. Can be used when you have an existing certificate and don't want a new certificate be generated automatically. If letsencrypt-domain is specified this property has no effect
      --cert-key string             Location of your SSL certificate private key. Can be used when you have an existing certificate and don't want a new certificate be generated automatically. If letsencrypt-domain is specified this property has no effect

Global Flags:
      --log-file string    sets Netbird log path. If console is specified the the log will be output to stdout (default "/var/log/netbird/signal.log")
      --log-level string    (default "info")

I don’t see a key that only the management would know about or some secret, it looks like the signal service is open on the internet and anyone is free to use it… yet when I try to use the public signal service from netbird cloud it does not work, so there has to be some sort of auth/filtering but I cannot find what.

And given various peers have to connect to it, I don’t see a way to restrict by IP either.

Have you tried these troubleshooting steps?

Reviewed client troubleshooting (if applicable)
Checked for newer NetBird versions
Searched for similar issues on GitHub (including closed ones)
Restarted the NetBird client
Disabled other VPN software
Checked firewall settings

saule1508 · July 13, 2025, 3:05am

I would think that when a peer connect to signal, it adds its public key in the metadata of the grpc connection, it is the way a client can add key/value for the grcp server to consume. You can see it here:

github.com/netbirdio/netbird

signal/client/grpc.go

3e6eede15


      
          
          func (c *GrpcClient) getStreamStatusChan() <-chan struct{} {
          	c.mux.Lock()
          	defer c.mux.Unlock()
          	if c.connectedCh == nil {
          		c.connectedCh = make(chan struct{})
          	}
          	return c.connectedCh
          }
          
          func (c *GrpcClient) connect(ctx context.Context, key string) (proto.SignalExchange_ConnectStreamClient, error) {
          	c.stream = nil
          
          	// add key fingerprint to the request header to be identified on the server side
          	md := metadata.New(map[string]string{proto.HeaderId: key})
          	metaCtx := metadata.NewOutgoingContext(ctx, md)
          	stream, err := c.realClient.ConnectStream(metaCtx, grpc.WaitForReady(true))
          	c.stream = stream
          	if err != nil {
          		return nil, err
          	}

I did not see where in the code the grpc server validates that this public key is part of the netbird network, but it does it for sure. Anyway each peer needs the public key of the other in order to decrypt the messages exchanged.
So signal is secured by the fact that runs on ssl and the clients authentify with their public key. Also the messages exchanged are encrypted using the wireguard keys (netbird/signal/client/grpc.go at 3e6eede1523f5073cd3d35f068c771b6d88ba397 · netbirdio/netbird · GitHub)
Signal does not contain any sensitive data, it is simply a mechanism used in the webrct protocol to allow two peers to establish a connection: each peer sends via signal the connection candidates about itself (discovered by webrtc, using STUN) to the other

PhilippeVaucher · July 13, 2025, 7:34pm

Thanks!

So that means that in management.json I should be able to use https://signal.netbird.io:443 (the cloud server) and it should work?

I tried and it does not.

Same behavior as with my own standalone signal server, it says “connected” but then all peers are relayed and nothing really works.

saule1508 · July 14, 2025, 4:27am

In the management.json of your self hosted instance you have to refer the signal of your self-hosted instance.
If you put the signal of another instance (the cloud one), I am pretty sure the cloud instance will reject the connections because the public key (which is used by the peer to authenticate when they do the grcp call) is not valid for that netbird network.
if you have relayed connections between the peers, it does not mean your signal is not working. You can look in the logs of the peers (put the log in debug) the candidates exchanged between them, this candidate exchange is done via signal, so for sure your signal is working

saule1508 · July 14, 2025, 6:47am

But actually you might be right, looking at the code I don’t see that the signal validates if the messages it receives from a peer belongs to this specific netbird instance. I think we would need two self-hosted instances, and then try to use the signal of the other, to be sure…

PhilippeVaucher · July 14, 2025, 7:35am

Thanks again.

Yes, for signal to be “part” of the netbird self hosted instance, the management & signal have to share some kind of user/password or secret key.

Right now it looks like signal just happily accepts anyone… except it does not work

Just to be clear, my current setup is one docker-compose with all services behind a traefik reverse proxy.

It works fine, but I had intermittent disconnections and thus I added several relays servers and made them standalone, and now it works great (they share NB_AUTH_SECRET with the management server).

I want to do the same for the signal service, but when I try netbird status says it is connected to the signal service, but netbird behaves strangely, none of my P2P connections seems to work anymore.

That said, I think I just had a “haha” moment, maybe it’s just I didn’t wait enough time and did not reconnect all my peers! I just reconnected my client but other peers still had the old signal server and thus couldn’t reach.

I’ll test again and report.