OpenSimulator Internals/Code Map/LLUDP

Overview

LLUDP is the network transport layer for viewer communication. It implements the Second Life UDP wire protocol -- packet framing, zero-encoding, reliable delivery with ACKs, resend handling, and per-client throttling. It sits below IClientAPI and LLClientView.

Source: OpenSim/Region/ClientStack/Linden/UDP/LLUDPServer.cs

Class hierarchy:

LLUDPServer → OpenSimUDPBase

One LLUDPServer instance exists per region, attached via LLUDPServerShim (an INonSharedRegionModule).

LLUDPServerShim

Thin adapter that lets LLUDPServer participate in the region module system.

AddRegion(scene) -- creates the LLUDPServer bound to the region's internal IP/port, sets scene.RegionInfo.InternalEndPoint.Port to whatever the server actually bound to
RegionLoaded(scene) -- calls Start()
RemoveRegion(scene) -- calls Stop()
AddScene(scene) -- registers ~10 StatsManager stats (client logouts, UDP receives/sends, packets processed/malformed/orphaned/resent, average process time)

Constants

MTU = 1400 bytes (maximum transmission unit for a single UDP packet)
MAXPAYLOAD = 1200 bytes

Construction

Config read from [ClientStack.LindenUDP]:

Setting	Default	Notes
client_socket_rcvbuf_size	0 (OS default)	Receive buffer size passed to the socket
scene_throttle_max_bps	6250000	Total bandwidth throttle for the whole region
TextureSendLimit	20	Texture packets queued per OnQueueEmpty trigger
DefaultRTO	0	Default retransmission timeout
MaxRTO	0	Maximum retransmission timeout
AckTimeout	60 (seconds)	Time without any packet before a client is disconnected
PausedAckTimeout	300 (seconds)	Extended timeout when client is paused (e.g. file upload dialog open)
DisableFacelights	false
SupportViewerObjectsCache	true

Constructor also measures the actual resolution of Environment.TickCount and Util.GetTimeStampMS() on this hardware -- logged at startup, used for timing accuracy elsewhere.

Start / Stop

Start() launches three things:

StartInbound() -- base class UDP receive loop, plus a dedicated "Incoming Packets (RegionName)" thread running IncomingPacketHandler()
StartOutbound() -- base class UDP send loop, plus a dedicated "Outgoing Packets (RegionName)" thread running OutgoingPacketHandler()
OqrEngine.Start() -- JobEngine for outgoing queue refills, decoupled from per-connection thread pool jobs to avoid performance problems with many connections

Stop() reverses all three.

Incoming Packet Path

PacketReceived(buffer)  -- called by OpenSimUDPBase for every UDP datagram

Decoding: validates minimum length (7 bytes), computes header length (7/8/10 bytes depending on extra header flags), zero-decodes if MSG_ZEROCODED flag set, calls Packet.BuildPacket()
Malformed packets are dropped and counted (IncomingMalformedPacketCount) -- logs a warning every 10000 as a probable network attack indicator
Packet-to-client mapping:
- If the endpoint has a pending queue (client still being created), the packet is queued rather than processed, except UseCircuitCode resends which are acked and dropped
- If no client exists yet and the packet is UseCircuitCode, a pending queue is created and HandleUseCircuitCode() is dispatched via FireAndForget
- Otherwise the client is looked up by endpoint via Scene.TryGetClient() -- unrecognized senders are counted as orphaned packets
ACK receiving: appended ACKs and standalone PacketAck packets are applied to udpClient.NeedAcks
ACK sending: reliable packets get their sequence number queued in PendingAcks; ACKs are flushed once 2*MTU bytes have been received since the last flush
Duplicate detection: PacketArchive tracks recently seen reliable sequence numbers -- duplicates (resent or not) are dropped after the ack machinery has already run
Ping handling: StartPingCheck/CompletePingCheck are handled inline and never reach the packet inbox
Everything else is wrapped in an IncomingPacket and added to packetInbox (a BlockingCollection) for the IncomingPacketHandler thread to process

IncomingPacketHandler() thread: pulls from packetInbox with a 4500ms timeout, calls client.ProcessInPacket() for each. Exceptions in one client's packet do not crash the loop.

Outgoing Packet Path

SendPacket() / SendPacketData()

Packets larger than MTU with variable blocks are split via packet.ToBytesMultiple() (CoarseLocationUpdate is exempt -- cannot be split)
Zero-encoding applied if MSG_ZEROCODED flag is set on the packet
Wrapped in an OutgoingPacket and either queued via udpClient.EnqueueOutgoing() or sent immediately via SendPacketFinal()

SendPacketFinal():

Appends pending ACKs to plain reliable packets if there's room (up to 256, then flags MSG_APPENDED_ACKS)
Assigns a new sequence number (unless this is a resend, which keeps its original data but sets MSG_RESENT)
Calls SyncSend() -- actual socket write
If the packet has no custom UnackedMethod, the buffer is freed immediately (fire-and-forget). Otherwise it's added to udpClient.NeedAcks awaiting acknowledgment

OutgoingPacketHandler() thread loop (runs continuously):

Every 100ms: triggers HandleUnacked() checks (resend expired packets)
Every 500ms (5 x 100ms): triggers SendAcks()
Every 5000ms (10 x 500ms): triggers SendPing()
Calls Scene.ForEachClient() with ClientOutgoingPacketHandler, which dequeues throttled outgoing packets per client
Sleeps 100ms if no clients connected, 15ms if nothing was sent this round (matches typical OS tick granularity), otherwise loops immediately

Reliability

Mechanism	Notes
NeedAcks	Per-client collection of sent reliable packets awaiting acknowledgment, with expiry based on RTO (retransmission timeout)
PendingAcks	Per-client queue of sequence numbers to be acknowledged back to the sender
PacketArchive	Per-client record of recently received reliable sequence numbers, used to detect and drop duplicates
ResendUnacked()	Sets MSG_RESENT flag, increments ResendCount, requeues or resends immediately
HandleUnacked()	Called every 100ms per client. Disconnects the client if no packet received within AckTimeout (or PausedAckTimeout if paused). Otherwise resends expired unacked packets.

Connection Establishment (HandleUseCircuitCode)

Triggered when a UseCircuitCode packet arrives from an endpoint with no existing client:

IsClientAuthorized() -- validates session/agent/circuit code via AgentCircuitManager.AuthenticateSession()
On success: AddClient() creates the LLUDPClient and LLClientView, starts the client
If AgentCircuitData is missing (shouldn't happen for a legitimately authorized circuit), the agent is force-closed
Any packets that arrived and were queued in m_pendingCache while the client was being created are reinjected into PacketReceived()
SendRegionHandshake() is called if this is not a teleport (teleportFlags <= 0)
On authorization failure: pending cache entry removed, warning logged, no client created

AddClient() is synchronized across the whole scene (lock (this)) to avoid race conditions -- referenced Mantis #5365 in source comments as the reason.

Client Disconnection

DeactivateClientDueToTimeout() -- called when HandleUnacked() detects no packets received within the timeout window. Logs a warning, then calls Scene.CloseAgent() (falls back to client.Close() if that fails).

LogoutHandler() -- sends the logout packet, then calls Scene.CloseAgent() if not already logging out.

Zero-Encoding

ZeroEncode() implements run-length encoding of zero bytes in outgoing packets (a bandwidth optimization from the original SL protocol) -- runs of zero bytes become a 0x00 marker followed by a count byte. Applied conditionally: if the zero-encoded result is not smaller than the original, the MSG_ZEROCODED flag is stripped and the packet is sent unencoded instead.

Notable Details

Malformed packet warnings fire every 10000 occurrences with the message "probable network attack" -- this is the closest thing to built-in DoS detection at this layer
The AddClient() lock is scene-wide, not per-agent -- a documented but still-present performance/correctness tradeoff (source comment references wanting to move to per-circuit locking eventually)
BinaryStats packet logging exists (LogPacketHeader) but is disabled by default -- controlled by [Statistics.Binary] Enabled
The outgoing loop's sleep constant (15ms) has a comment explicitly referencing Windows' ~16ms scheduler granularity to avoid the OS rounding up to 32ms