OpenSimulator Internals/Code Map/LLUDP
OpenSimulator Internals/Code Map/LLUDP
[edit]Overview
[edit]LLUDP is the network transport layer for viewer communication. It implements the Second Life UDP wire protocol -- packet framing, zero-encoding, reliable delivery with ACKs, resend handling, and per-client throttling. It sits below IClientAPI and LLClientView.
Source: OpenSim/Region/ClientStack/Linden/UDP/LLUDPServer.cs
Class hierarchy:
LLUDPServer → OpenSimUDPBase
One LLUDPServer instance exists per region, attached via LLUDPServerShim (an INonSharedRegionModule).
LLUDPServerShim
[edit]Thin adapter that lets LLUDPServer participate in the region module system.
- AddRegion(scene) -- creates the LLUDPServer bound to the region's internal IP/port, sets scene.RegionInfo.InternalEndPoint.Port to whatever the server actually bound to
- RegionLoaded(scene) -- calls Start()
- RemoveRegion(scene) -- calls Stop()
- AddScene(scene) -- registers ~10 StatsManager stats (client logouts, UDP receives/sends, packets processed/malformed/orphaned/resent, average process time)
Constants
[edit]- MTU = 1400 bytes (maximum transmission unit for a single UDP packet)
- MAXPAYLOAD = 1200 bytes
Construction
[edit]Config read from [ClientStack.LindenUDP]:
| Setting | Default | Notes |
|---|---|---|
| client_socket_rcvbuf_size | 0 (OS default) | Receive buffer size passed to the socket |
| scene_throttle_max_bps | 6250000 | Total bandwidth throttle for the whole region |
| TextureSendLimit | 20 | Texture packets queued per OnQueueEmpty trigger |
| DefaultRTO | 0 | Default retransmission timeout |
| MaxRTO | 0 | Maximum retransmission timeout |
| AckTimeout | 60 (seconds) | Time without any packet before a client is disconnected |
| PausedAckTimeout | 300 (seconds) | Extended timeout when client is paused (e.g. file upload dialog open) |
| DisableFacelights | false | |
| SupportViewerObjectsCache | true |
Constructor also measures the actual resolution of Environment.TickCount and Util.GetTimeStampMS() on this hardware -- logged at startup, used for timing accuracy elsewhere.
Start / Stop
[edit]Start() launches three things:
- StartInbound() -- base class UDP receive loop, plus a dedicated "Incoming Packets (RegionName)" thread running IncomingPacketHandler()
- StartOutbound() -- base class UDP send loop, plus a dedicated "Outgoing Packets (RegionName)" thread running OutgoingPacketHandler()
- OqrEngine.Start() -- JobEngine for outgoing queue refills, decoupled from per-connection thread pool jobs to avoid performance problems with many connections
Stop() reverses all three.
Incoming Packet Path
[edit]PacketReceived(buffer) -- called by OpenSimUDPBase for every UDP datagram
- Decoding: validates minimum length (7 bytes), computes header length (7/8/10 bytes depending on extra header flags), zero-decodes if MSG_ZEROCODED flag set, calls Packet.BuildPacket()
- Malformed packets are dropped and counted (IncomingMalformedPacketCount) -- logs a warning every 10000 as a probable network attack indicator
- Packet-to-client mapping:
- If the endpoint has a pending queue (client still being created), the packet is queued rather than processed, except UseCircuitCode resends which are acked and dropped
- If no client exists yet and the packet is UseCircuitCode, a pending queue is created and HandleUseCircuitCode() is dispatched via FireAndForget
- Otherwise the client is looked up by endpoint via Scene.TryGetClient() -- unrecognized senders are counted as orphaned packets
- ACK receiving: appended ACKs and standalone PacketAck packets are applied to udpClient.NeedAcks
- ACK sending: reliable packets get their sequence number queued in PendingAcks; ACKs are flushed once 2*MTU bytes have been received since the last flush
- Duplicate detection: PacketArchive tracks recently seen reliable sequence numbers -- duplicates (resent or not) are dropped after the ack machinery has already run
- Ping handling: StartPingCheck/CompletePingCheck are handled inline and never reach the packet inbox
- Everything else is wrapped in an IncomingPacket and added to packetInbox (a BlockingCollection) for the IncomingPacketHandler thread to process
IncomingPacketHandler() thread: pulls from packetInbox with a 4500ms timeout, calls client.ProcessInPacket() for each. Exceptions in one client's packet do not crash the loop.
Outgoing Packet Path
[edit]SendPacket() / SendPacketData()
- Packets larger than MTU with variable blocks are split via packet.ToBytesMultiple() (CoarseLocationUpdate is exempt -- cannot be split)
- Zero-encoding applied if MSG_ZEROCODED flag is set on the packet
- Wrapped in an OutgoingPacket and either queued via udpClient.EnqueueOutgoing() or sent immediately via SendPacketFinal()
SendPacketFinal():
- Appends pending ACKs to plain reliable packets if there's room (up to 256, then flags MSG_APPENDED_ACKS)
- Assigns a new sequence number (unless this is a resend, which keeps its original data but sets MSG_RESENT)
- Calls SyncSend() -- actual socket write
- If the packet has no custom UnackedMethod, the buffer is freed immediately (fire-and-forget). Otherwise it's added to udpClient.NeedAcks awaiting acknowledgment
OutgoingPacketHandler() thread loop (runs continuously):
- Every 100ms: triggers HandleUnacked() checks (resend expired packets)
- Every 500ms (5 x 100ms): triggers SendAcks()
- Every 5000ms (10 x 500ms): triggers SendPing()
- Calls Scene.ForEachClient() with ClientOutgoingPacketHandler, which dequeues throttled outgoing packets per client
- Sleeps 100ms if no clients connected, 15ms if nothing was sent this round (matches typical OS tick granularity), otherwise loops immediately
Reliability
[edit]| Mechanism | Notes |
|---|---|
| NeedAcks | Per-client collection of sent reliable packets awaiting acknowledgment, with expiry based on RTO (retransmission timeout) |
| PendingAcks | Per-client queue of sequence numbers to be acknowledged back to the sender |
| PacketArchive | Per-client record of recently received reliable sequence numbers, used to detect and drop duplicates |
| ResendUnacked() | Sets MSG_RESENT flag, increments ResendCount, requeues or resends immediately |
| HandleUnacked() | Called every 100ms per client. Disconnects the client if no packet received within AckTimeout (or PausedAckTimeout if paused). Otherwise resends expired unacked packets. |
Connection Establishment (HandleUseCircuitCode)
[edit]Triggered when a UseCircuitCode packet arrives from an endpoint with no existing client:
- IsClientAuthorized() -- validates session/agent/circuit code via AgentCircuitManager.AuthenticateSession()
- On success: AddClient() creates the LLUDPClient and LLClientView, starts the client
- If AgentCircuitData is missing (shouldn't happen for a legitimately authorized circuit), the agent is force-closed
- Any packets that arrived and were queued in m_pendingCache while the client was being created are reinjected into PacketReceived()
- SendRegionHandshake() is called if this is not a teleport (teleportFlags <= 0)
- On authorization failure: pending cache entry removed, warning logged, no client created
AddClient() is synchronized across the whole scene (lock (this)) to avoid race conditions -- referenced Mantis #5365 in source comments as the reason.
Client Disconnection
[edit]DeactivateClientDueToTimeout() -- called when HandleUnacked() detects no packets received within the timeout window. Logs a warning, then calls Scene.CloseAgent() (falls back to client.Close() if that fails).
LogoutHandler() -- sends the logout packet, then calls Scene.CloseAgent() if not already logging out.
Zero-Encoding
[edit]ZeroEncode() implements run-length encoding of zero bytes in outgoing packets (a bandwidth optimization from the original SL protocol) -- runs of zero bytes become a 0x00 marker followed by a count byte. Applied conditionally: if the zero-encoded result is not smaller than the original, the MSG_ZEROCODED flag is stripped and the packet is sent unencoded instead.
Notable Details
[edit]- Malformed packet warnings fire every 10000 occurrences with the message "probable network attack" -- this is the closest thing to built-in DoS detection at this layer
- The AddClient() lock is scene-wide, not per-agent -- a documented but still-present performance/correctness tradeoff (source comment references wanting to move to per-circuit locking eventually)
- BinaryStats packet logging exists (LogPacketHeader) but is disabled by default -- controlled by [Statistics.Binary] Enabled
- The outgoing loop's sleep constant (15ms) has a comment explicitly referencing Windows' ~16ms scheduler granularity to avoid the OS rounding up to 32ms