Perceive-then-Plan: Layout-as-Policy for Monocular 3D Scene Layout Estimation

AbstractFigure 1

Layout estimation as a sequence of decisions

One-shot regressors predict every box at once, so a single mistake can break support and contact for the whole room. We perceive first, then plan corrections.

The Perceiver grounds all instances in 3D from the image and 2D boxes; the LaP Planner then refines the canonical layout with steps such as SELECT obj 1, MOVE [1,0,0], STOP. — Fig. 1The Perceiver grounds every instance in 3D. The LaP Planner then refines the scene with action sequences, following the Layout-as-Policy design.

Building structured 3D scene layouts from a single image requires reconciling visual observations with physical and spatial constraints, a challenge that is difficult to address with direct prediction alone. In this work, we formulate monocular 3D layout estimation as a perceive-then-plan problem with vision-language models, where a Perceiver first grounds the 3D objects and then a Planner iteratively refines the scene hypothesis through actions that improve physical plausibility while preserving consistency with the input image.

We propose Layout-as-Policy (LaP), which casts the planning stage as a policy learning problem: 3D layouts are represented as structured states, and refined via discrete actions such as translation, rotation, and rescaling. Starting from an observation-aligned initialization with the geometry-enhanced Perceiver, the LaP Planner is trained to produce action sequences that progressively resolve geometric inconsistencies and enforce realistic spatial relations. To enable effective learning, we combine supervised sequence initialization with preference-based optimization, allowing the model to learn corrective behaviors without requiring explicit reward engineering.

This formulation transforms layout estimation from a one-shot prediction task into an iterative refinement process, enabling better handling of global constraints and complex object interactions. Experiments demonstrate that our approach produces layouts that are more physically coherent and better aligned with visual observations, while naturally supporting downstream tasks such as scene editing and manipulation.

§3 MethodFigure 2

Two VLMs, one closed loop

The Perceiver supplies a visually grounded starting point. The LaP Planner corrects its physical and relational errors. Pick a stage to see what it does.

Pipeline: Perceive stage with a 2D detector, a frozen geometry encoder and the Perceiver VLM; Plan stage where the LaP Planner with LoRA reads the scene state, the 2D detections and an instruction, outputs an action sequence that the Action Executer applies; Formatting output with scene assembly and aligned 3D grounding. — Fig. 2The Perceiver grounds 3D boxes from the input image. A canonicalized, grid-based representation is then iteratively refined by the LaP Planner, which treats each layout as a structured state and selects discrete actions until convergence. The final layout supports both scene assembly (digital twin) and camera-space projection (3D grounding).

Local-axis box representation

Regressing eight box vertices needs strong format priors that do not suit a VLM's training objective. Each box is instead a center, a size, and its two bottom axes; the vertical axis follows from a cross product.

ax_y = ax_x × ax_z
Box = {(x, y, z, w, h, l, ax_x, ax_z, cls)}

Without it: Reproj. IoU 0.54 → 0.41

Geometry-aware feature modulation

Fine-tuning alone fixes the output format but barely improves 3D grounding. Features from a frozen VGGT encoder regress FiLM coefficients and a spatial gate that inject geometry into the VLM's visual tokens. Thanks to the residual form, the fused features start out close to the 2D ones.

F = g ⊙ [(1+Δγ) ⊙ F3d + β] + (1−g) ⊙ F2d
g = σ(MLP(F3d))

Removing it as well: Avg. DE 0.10 → 0.18 m

Grid-based canonical layout

Gravity is the dominant axis from an SVD of all bottom-face normals. The scene is rotated into a gravity-aligned frame, centered on the ground plane, and discretized into integer grid indices.

g  = round((v − o) / δ)     δ = 0.1 m
gθ = round((θ + π) / (2π/Nθ))   Nθ = 24

1 cell = 10 cm · yaw in 15° bins

The layout is a state

Each object is written out as text: id, class, its 2D box (0–1000), grid position, size, and yaw index. The Planner sees this state together with the image.

obj_0 cabinet
  pos: [gx, gy, gz]   size: [gw, gh, gl]
  yaw: 0–23

Refinement is a policy

The LaP Planner (Qwen3-VL with LoRA) predicts an action sequence. An action executor applies it to produce the next state, which is fed back with the same image.

L(t+1) = execute(L(t), π(I, L(t)))

Several short rounds

Fixing every inconsistency at once is a combinatorial problem. Each round resolves a few violations, and refinement stops when the Planner outputs only <STOP> or hits the round limit.

LaP 8B collisions: 3.62 → 3.23 after 1 round, 1.44 at convergence

Contact graph

Real assets rarely match their boxes exactly. With the LoRA adapter disabled, the same Planner backbone labels every object as resting on the FLOOR, ON another object, or FREE (wall-mounted or hanging).

<CONTACT> id: 2 class: pillow relation: ON 3 </CONTACT>

Relational bundles

A supporter and everything stacked on it move as one rigid composite, so a pillow never drifts away from its bed. FREE objects keep their refined position.

Gravity settling

A single-pass rigid-body simulation under a floor constraint removes the floating and penetration artifacts left by asset–box mismatch. The result is a digital twin, or boxes projected back to camera space for 3D grounding.

§3.2 Layout-as-PolicyFigure 3

Five actions, learned from preferences

A compact action space covers the Perceiver's dominant failure modes: misplaced position, wrong height, wrong scale and wrong orientation. Judging which of two action sequences is better is easier than defining the single best one, so the Planner learns from preference pairs instead of a hand-made reward.

action space · integers only

SELECT obj_N
Choose which object the following actions edit.
MOVE [dx, dy, dz]
Shift along the ground plane (dx, dz) and vertically (dy), in 10 cm grid cells.
ROTATE_Y [d]
Turn around the vertical axis by d × 15°.
RESIZE [d]
Rescale all three box dimensions uniformly.
STOP
End the current sequence. A lone STOP means the layout is done.

Rules: select before correcting, fix the largest errors first, prefer fewer actions.

Synthesize perturbed layouts

Start from a ground-truth layout, sample an object and apply random, non-repeating perturbations from the action space; continue with probability p. Inverting the perturbations gives the ground-truth corrective sequence.
SFT warm-up

Supervised fine-tuning on these pairs teaches the output format and a basic reading of action, layout and image together, before the Planner ever sees noisy Perceiver outputs.
Direct preference optimization

The ground-truth sequence is the chosen sample. Perceiver-initialized trajectories and degraded copies of the ground truth are rejected samples.

Numerical shiftRight direction, wrong magnitude.

Nuisance actionsNo-ops, duplicates, edits to correct objects.

Missing actionsPartial fixes or a premature STOP.

Wrong typesMOVE swapped for ROTATE_Y, or flipped signs.

Planner training mix (SFT and DPO)

20% ground truth, answer is STOP
30% Perceiver-initialized
50% synthetic perturbations

DPO data synthesis: a ground-truth layout is perturbed with sampled actions; the inverted sequence is the selected sample and degraded versions are rejected samples.

Fig. 3DPO pair synthesis. Perturbing a ground-truth layout and inverting the perturbation yields the selected sequence; degrading that sequence yields rejected ones. Sequences can touch multiple objects.

Perturbations create the same kinds of conflict the Perceiver produces, such as collisions and broken support, so the Planner learns visual alignment and physical plausibility in the same training phase. For DPO pairs built from Perceiver outputs, a candidate is kept as preferred only if it clearly improves layout-quality metrics on the same set of objects; ambiguous pairs are discarded.

§4.2 QualitativeFigure 5

One round, one short program

Each round the Planner looks at the image and the current layout, then writes a few lines of actions. Circled objects are the ones it chose to fix.

Two scenes: reference image, layouts after each round with the predicted actions, the refined layout, and the recovered 3D scene. — Fig. 5LaP-guided refinement. Each column shows the structured layout of one round with the action sequence the Planner predicted for it, followed by the refined layout and the recovered scene.

Supplementary360° renders

Every layout, from every side

Turntable renders of refined 3D layouts, each estimated from the single photo in the inset. Seen from behind and from the side, objects keep their support and spacing. Click a view to pause it.

Input photo of a hotel room with a desk, lamp, armchair and a television on a dresser — **Hotel room**lamp, chairs, tables, television, door

Input photo of an office corner with a wooden desk, a CRT monitor, a lamp, a chair and a bookcase — **Office**desks, monitor, lamp, chair, bookcase

Input photo of a furniture-store bedroom with a bed, dressers, lamps and a small table with chairs — **Bedroom**bed, pillows, cabinets, lamps, mirror, chairs

Input photo of a classroom with long tables, a blue sofa, a wall-mounted television and a board — **Classroom**sofas, tables, television, board, counter

§4 ExperimentsTables 1–4 · Figure 4

Better grounding, then fewer physical violations

Trained on Hypersim and SUN RGB-D, evaluated on 1,000 SUN RGB-D test images. The Perceiver is compared with open and closed VLMs. The Planner is compared before and after refinement, since no prior VLM method refines 3D layouts.

**Table 1**3D grounding of the Perceiver on 1,000 SUN RGB-D test images. Bold: best, underlined: second best. † direct 3D grounding, ‡ plain supervised fine-tuning.
Model	Reproj. IoU ↑	Prec@0.25 ↑	Prec@0.5 ↑	Avg. DE (m) ↓
Closed-source
Gemini-2.5-Flash †	0.24	0.19	0.07	0.47
GPT-4o †	0.27	0.19	0.08	0.55
Open-source
LLaMA3.2-Vision 11B	0.14	0.18	0.09	0.54
Qwen3-VL 8B	0.18	0.31	0.13	0.43
Qwen3-VL 8B ‡	0.44	0.84	0.46	0.18
Qwen3-VL 8B †	0.31	0.54	0.28	0.25
Qwen3-VL 32B	0.26	0.32	0.18	0.37
Qwen3-VL 32B †	0.33	0.57	0.29	0.21
InternVL3.5 8B	0.13	0.24	0.11	0.58
Gemma3 27B	0.24	0.23	0.14	0.47
Gemma3 27B †	0.30	0.22	0.15	0.55
GLM-4.6V-Flash 9B	0.21	0.28	0.13	0.36
N3D-VLM 7B †	0.33	0.53	0.35	0.19
Ours
Perceiver 2B	0.54	0.91	0.62	0.09
Perceiver 8B	0.58	0.90	0.64	0.08

3D grounding comparison on two scenes: Qwen3-VL 32B, Qwen3-VL 8B with SFT, N3D-VLM 7B and our Perceiver 8B, with per-image metrics. — Fig. 43D grounding by the Perceiver 8B against strong baselines. Boxes stay tight for objects with non-trivial orientation and partial occlusion.

**Table 2**Action-based refinement, all starting from the Perceiver 2B layout. Green marks a gain and red a degradation relative to Perceiver only.
Model	State	Reproj. IoU ↑	SVR (%) ↓	# Collisions ↓	Rot. Err. ↓	Avg. DE ↓
Perceiver only	Before ref.	0.54	11.63	3.62	17.74	0.09
One-shot VLM direct ref.	After ref.	0.55+0.01	10.48−1.15	3.54−0.08	14.83−2.91	0.11+0.02
Rule-based ref.	After ref.	0.26−0.28	0.00−11.63	0.00−3.62	17.74±0.00	0.24+0.15
GPT-4o	After ref. (converge)	0.49−0.05	9.96−1.67	3.89+0.27	13.35−4.39	0.13+0.04
LaP Planner (2B)	After ref. (1 iter)	0.57+0.03	8.81−2.82	3.21−0.41	15.25−2.49	0.09±0.00
LaP Planner (2B)	After ref. (converge)	0.65+0.11	4.85−6.78	1.46−2.16	9.05−8.69	0.07−0.02
LaP Planner (8B)	After ref. (1 iter)	0.57+0.03	8.62−3.01	3.23−0.39	14.37−3.37	0.09±0.00
LaP Planner (8B)	After ref. (converge)	0.67+0.13	4.12−7.51	1.44−2.18	8.62−9.12	0.07−0.02

**Table 3**Perceiver ablation. Rows remove components cumulatively.
Configuration	Reproj. IoU ↑	Prec@0.25 ↑	Prec@0.5 ↑	Avg. DE ↓
Full model (2B)	0.54	0.91	0.62	0.09
− Local-axis	0.41−0.13	0.79−0.12	0.48−0.14	0.10+0.01
− Geo. modulation	0.36−0.05	0.65−0.14	0.46−0.02	0.18+0.08

Dropping geometry modulation on top of the local-axis representation nearly doubles depth error, from 0.10 m to 0.18 m.

**Table 4**Planner ablation, LaP Planner 2B at convergence.
Configuration	IoU ↑	SVR % ↓	Coll. ↓	Rot. ↓	Actions
Full model (2B)	0.65	4.85	1.46	9.05	3.67
w/o grid rep.	0.58−0.07	5.29+0.42	2.43+0.97	15.94+6.89	3.53
w/o SFT warm-up	0.56−0.09	5.12+0.27	2.18+0.72	13.43+4.38	3.96
w/o DPO	0.60−0.05	8.77+3.88	3.02+1.56	13.98+4.93	4.73

DPO teaches the Planner to compare competing sequences, producing minimal but sufficient actions per round.

More resultsFigures 6, 7, 10, 11

Gallery

Grounding, refinement and assembled scenes across cluttered rooms, varied object scales and heavy occlusion. Click any figure to enlarge.

Grid of indoor photos with 3D boxes predicted by the Perceiver, plus two canonical layouts of chairs and tables. — Fig. 10Perceiver results on real-world captures, shown as reprojection and as layout in 3D space.

§4.4 ApplicationsFigures 8, 9

The action space is also an editing interface

Because the Planner speaks in actions, a user instruction becomes an action sequence over the same vocabulary. No new architecture is needed.

Language-guided editing

Instructions are translated into actions that change the layout while keeping it plausible. The edited layout and the recovered scene are shown together.

Instruction“Put bottle_0 right above stove_0”

Instruction“Put the stove on counter_1”

Kitchen: input image, refined 3D layout, and two edited layouts with recovered scenes for the two instructions. — Fig. 8Scene editing with instructions.

Instructional refinement

A user can point out what looks wrong, and the Planner answers with the corrective actions. This makes it a natural component for interactive agentic systems.

User“The sofa_0 and the table_0 are too close to each other, should be more separated”

LaP Planner<SELECT> sofa_0, <MOVE> (0, 0, 4), <STOP>

Classroom: the sofa and table overlap before the instruction and are separated after the predicted MOVE. — Fig. 9Instructional refinement from user language.

Citation

BibTeX

@inproceedings{zhou2026perceivethenplan,
  title={Perceive-then-Plan: Layout-as-Policy for Monocular 3D Scene Layout Estimation},
  author={Zhou, Junwei and Tai, Yu-Wing},
  booktitle={Advances in Neural Information Processing Systems},
  year={2026}
}

Perceive ‑then‑Plan