feat: implement automated 3D character pipeline with retargeting and rig management tools
This commit is contained in:
Nicholas Butzke
+195
-95
@@ -1,127 +1,227 @@
|
||||
# Character Pipeline: Sketchfab → Rigged → Animated → In-Game
|
||||
# Character Pipeline: source model → rigged → animated → in-game
|
||||
|
||||
One command turns a Sketchfab model into a playable character skin:
|
||||
One command turns a model into a playable character skin:
|
||||
|
||||
```bash
|
||||
python tools/pipeline.py --uid <sketchfab-uid> --name space_marine
|
||||
```
|
||||
|
||||
That downloads the model, auto-rigs it in Blender, merges the shared
|
||||
animation library onto it, writes `assets/characters/skins/space_marine.glb`,
|
||||
and registers it in `skins.json`. Restart the game — the skin is in the main
|
||||
menu dropdown, fully animated in first AND third person, synced in multiplayer.
|
||||
That downloads the model, moves the shared animation library onto it, writes
|
||||
`assets/characters/skins/space_marine.glb` plus a `.rig.json` sidecar, verifies
|
||||
the result, and registers it in `skins.json`. Restart the game — the skin is in
|
||||
the main menu dropdown, fully animated in first AND third person, synced in
|
||||
multiplayer.
|
||||
|
||||
## Prerequisites (one-time setup)
|
||||
## The one rule: keep the model's own rig
|
||||
|
||||
1. **Sketchfab API token** — from <https://sketchfab.com/settings/password>.
|
||||
Put it in the env var `SKETCHFAB_API_TOKEN`, or in a file named
|
||||
`.sketchfab_token` in the project root (gitignored).
|
||||
2. **Blender 3.6+** — on PATH, or set `BLENDER_PATH` to `blender.exe`.
|
||||
3. **Animation library** — already bundled. `assets/characters/animations/_library.glb`
|
||||
is the CC0 Quaternius Universal Animation Library (13 game-relevant clips
|
||||
mapped in `LIBRARY_CLIP_MAP`). `tools/autorig.py` rigs every character to
|
||||
this library's skeleton (fitting the arm bones to the model's actual pose),
|
||||
and `merge_animations.py` retargets each clip onto that fitted rest. To use a
|
||||
different/larger set instead, see "Swapping the animation library" below.
|
||||
**If a model arrives with a skeleton, that skeleton is what ships.** Its bones,
|
||||
its artist-painted weights, its separate per-part meshes, and its dedicated
|
||||
skirt/hair bone chains all survive; only the ANIMATION is moved onto it.
|
||||
|
||||
This is the whole point, and the pipeline used to do the opposite. The old route
|
||||
(`strip_rig.py` → `autorig.py` → `merge_animations.py`) discarded any foreign
|
||||
skeleton, joined every mesh into one, and rebound the result by weighting each
|
||||
vertex to its nearest four bone segments. It did that to work around a *naming*
|
||||
problem — the retarget matched bones by exact name — and the cost was the entire
|
||||
asset. Measured on the shipped `taila.glb` against the source it was built from:
|
||||
|
||||
| | source model | old pipeline output |
|
||||
|---|---|---|
|
||||
| vertices pulled by BOTH legs | 17 (0.1%) | **2817 (16%)**, worst a dead 50/50 |
|
||||
| vertices at the full 4 influences | 26% | **86%** |
|
||||
| meshes (body / cloth / hair separable) | 18 | **1** |
|
||||
| skirt bone chains | 21 bones | **0** |
|
||||
| hair bone chains | ~50 bones | **0** |
|
||||
| limb twist bones | 8 | **0** |
|
||||
|
||||
A vertex pulled equally by both legs sits between them and stays there while
|
||||
they separate, stretching every triangle around it — that is the squashing and
|
||||
the "elongated boot". `characters/skin_leg_repair.gd` exists solely to undo this
|
||||
at load time, by snapping weights and deleting triangles on a mesh it has to
|
||||
rebuild every spawn.
|
||||
|
||||
The naming problem is now solved properly, in `tools/rig_map.py`, so nothing has
|
||||
to be thrown away.
|
||||
|
||||
## The pipeline, step by step
|
||||
|
||||
Each step is also runnable on its own:
|
||||
|
||||
| Step | Tool | What it does |
|
||||
|---|---|---|
|
||||
| 1. Find | `python tools/sketchfab_import.py search "anime robot" --rigged` | Search downloadable models (license shown per result) |
|
||||
| 2. Download | `python tools/sketchfab_import.py download <uid>` | GLB + license/attribution JSON into `assets/characters/incoming/` |
|
||||
| 3. Auto-rig | `blender --background --python tools/autorig.py -- in.glb out.glb` | Fits the skeleton (incl. arm bones) to the mesh, binds smooth nearest-bone weights |
|
||||
| 4. Animate | `blender --background --python tools/merge_animations.py -- rigged.glb assets/characters/animations out.glb` | Retargets the clip library onto the fitted rest, strips root motion, canonical names |
|
||||
| 5. Register | (automatic in pipeline.py) | Copies to `skins/`, adds entry to `skins.json` |
|
||||
| 2. Download | `python tools/sketchfab_import.py download <uid>` | GLB + license JSON into `assets/characters/incoming/` |
|
||||
| 3. Retarget | `blender --background --python tools/retarget.py -- in.glb assets/characters/animations out.glb` | Keeps the rig; moves the clip library onto it |
|
||||
| 3b. *(unrigged only)* | `blender --background --python tools/autorig.py -- in.glb rigged.glb` | Fits the library skeleton and solves weights — lossy, see below |
|
||||
| 4. Verify | `blender --background --python tools/verify_character.py -- out.glb` | Gates the build on the defects listed below |
|
||||
| 5. Register | (automatic in `pipeline.py`) | Copies to `skins/`, adds an entry to `skins.json` |
|
||||
|
||||
`tools/pipeline.py` chains all of it. Useful flags:
|
||||
`pipeline.py` chains all of it and picks the path automatically — it reads the
|
||||
glTF container to see whether a `skins` array is present. Useful flags:
|
||||
|
||||
- `--input file.glb` instead of `--uid` for local files (GLB/FBX/OBJ).
|
||||
- `--rigged` — skip auto-rig for models that already have a skeleton
|
||||
(Mixamo/AccuRig/Tripo output). Bones get renamed to Mixamo convention
|
||||
automatically when recognizable.
|
||||
- `--height 1.6` — target character height in meters.
|
||||
- `--input file.glb` instead of `--uid` for local files.
|
||||
- `--rigged` — force the keep-the-rig path (needed for FBX, which cannot be probed).
|
||||
- `--rebind` — force the lossy path. Last resort.
|
||||
- `--height 1.6` — target character height in metres.
|
||||
|
||||
## Auto-rigging: what to use when
|
||||
### How the retarget works
|
||||
|
||||
Mixamo has **no public API** (and Adobe has said one isn't coming), so full
|
||||
automation needs an alternative. In order of preference:
|
||||
`tools/rig_map.py` resolves both skeletons to ROLES and pairs them up, so bone
|
||||
names never have to match. It works structurally wherever a name would lie:
|
||||
|
||||
1. **Built-in autorig (`tools/autorig.py`)** — heuristic skeleton fit +
|
||||
Blender automatic weights, fully automated, zero cost. Works well for
|
||||
normal-proportioned upright humanoids. Weakest on flowing
|
||||
dresses/capes/extreme proportions.
|
||||
2. **Mixamo web (manual, ~2 min/model)** — upload FBX/OBJ at
|
||||
<https://www.mixamo.com>, place 7 markers, download rigged FBX "without
|
||||
animations", then `pipeline.py --input rigged.fbx --rigged --name x`.
|
||||
Best weight quality for the effort.
|
||||
3. **Reallusion AccuRig** (free desktop app) — better auto-weights than
|
||||
Mixamo for tricky meshes; export FBX and feed with `--rigged`.
|
||||
4. **UniRig / Tripo AI / Meshy / Anything World** — ML auto-riggers. UniRig is
|
||||
open source (VAST/Tripo); Tripo, Meshy, and Anything World offer paid HTTP
|
||||
APIs if you ever want a fully hosted rigging step with no Blender at all.
|
||||
- **Hips** is found as the base of the longest non-limb, non-cosmetic chain —
|
||||
not by looking for "hips". Rigify calls it `DEF-spine`.
|
||||
- **The head** is wherever that chain ends. A stock Rigify rig has no bone with
|
||||
"head" in its name at all; the head is `DEF-spine.006`.
|
||||
- **Chains of different lengths** are matched by normalised position, so a
|
||||
6-bone spine is driven by a 5-bone one.
|
||||
|
||||
All roads lead to the same place: a GLB with Mixamo bone names, which is what
|
||||
`merge_animations.py` and the game expect.
|
||||
`tools/retarget.py` then bakes each clip as a rest-relative delta —
|
||||
`R_world = src_pose · src_rest⁻¹`, applied as `R_world · tgt_rest` — rather than
|
||||
copying absolute world orientation, which would force the library's bone roll
|
||||
onto a mesh bound with a different one and twist every limb by a constant offset.
|
||||
|
||||
It also **rebuilds parenting**. A Rigify DEF-rig exports its chain roots
|
||||
parented straight to the armature root (Rigify drives them by constraint, not
|
||||
hierarchy), so on import the thighs, skirt and hair all hang off the root and
|
||||
would float in place while the body moves. Orphans are re-attached by anatomy
|
||||
where it is known and by rest geometry otherwise. **Cloth may only ever attach
|
||||
to the trunk, never to a limb** — anchor a skirt panel to the nearest bone and
|
||||
16 of Taila's 21 land on a thigh, where the panel rides one leg like a trouser
|
||||
leg.
|
||||
|
||||
### What is deliberately NOT driven
|
||||
|
||||
Skirt, hair, twist and face bones get **no animation tracks at all**. They rest
|
||||
relative to their parents and belong to the runtime instead. That split — clips
|
||||
animate the body, physics animates the cloth — is what makes clothes read as
|
||||
clothes, and it is why the exported clips only carry the ~53 bones they need
|
||||
(`taila.glb` went from 3.5 MB to 2.9 MB even after regaining its textures).
|
||||
|
||||
## Secondary motion (`characters/spring_bones.gd`)
|
||||
|
||||
Skin weights can only ever make a garment a rigid shell of whatever it is
|
||||
weighted to: weight a skirt to the thighs and it becomes trousers, weight it to
|
||||
the hips and it becomes a bell that never moves. Neither is cloth. A skirt is
|
||||
cloth because it LAGS. That is inertia, and it has to be integrated, not skinned.
|
||||
|
||||
Each cloth bone is a damped spring holding its tip toward where rigidly
|
||||
following its parent would have put it, then pinned to the bone's length and
|
||||
pushed out of the leg capsules so a skirt swings AROUND a thigh rather than
|
||||
through it. Bone tip directions and capsule radii are **measured from the
|
||||
model's own geometry at build time** and stored in the sidecar — a glTF skeleton
|
||||
carries no bone tails, and Taila's skirt bones have no children either, so
|
||||
nothing in the skeleton says which way a panel hangs or how thick a thigh is.
|
||||
|
||||
Tuning per class (hair stiffer and lighter, skirt slacker and heavier) lives in
|
||||
`SpringBones.TUNING`.
|
||||
|
||||
### `<model>.rig.json`
|
||||
|
||||
Written next to every built GLB, so the runtime never re-guesses anatomy:
|
||||
|
||||
- `roles` — resolved bone per role. `ShooterPoseModifier` reads this; without it
|
||||
its hips/neck/head lookups silently missed on any rig that names them
|
||||
differently, and every lean and aim-pitch did nothing.
|
||||
- `chains` — cloth/hair chains, with a measured tip vector per bone.
|
||||
- `colliders` — leg capsules with measured radii.
|
||||
- `twist` — limb twist bones.
|
||||
- `weights_authored` — **measured**, not inferred from which branch ran. This is
|
||||
what decides whether `SkinLegRepair` runs at load time. A model that arrives
|
||||
unrigged still gets a sidecar, and its solved weights still need the repair.
|
||||
|
||||
Raw `.glb` **and `.json`** must be in the export include filter, or the sidecar
|
||||
is missing from a build and every character loses its cloth.
|
||||
|
||||
## When a model has no skeleton
|
||||
|
||||
Then there is no authored weighting to keep and `autorig.py` fits the library
|
||||
skeleton with nearest-bone weights. This is genuinely lossy and the verifier
|
||||
reports it as warnings rather than failures, because no better result is
|
||||
available:
|
||||
|
||||
```
|
||||
[WARN] cross-leg blending is limited to draping cloth — 590 verts (21.2%)
|
||||
[WARN] influences look authored, not solved — 78% of verts carry 4 influences
|
||||
[WARN] model keeps its per-part meshes — 1 meshes
|
||||
```
|
||||
|
||||
`miku` is such a model. `SkinLegRepair` stays on for it at load time.
|
||||
|
||||
Prefer, in order: a model that ships rigged → **Mixamo web** (upload FBX/OBJ,
|
||||
place 7 markers, download rigged "without animations") → **Reallusion AccuRig**
|
||||
→ UniRig / Tripo / Meshy. All of them produce a rig this pipeline will keep.
|
||||
|
||||
## Materials
|
||||
|
||||
Anime models are very often exported "unlit": `KHR_materials_unlit`, a **black**
|
||||
`baseColorFactor`, and the real texture wired to `emissiveTexture`. Renderers
|
||||
honouring the unlit extension use base colour and ignore emission — so Blender
|
||||
reads black, never references the images, and imports with `bpy.data.images`
|
||||
*empty*. The character comes out a silhouette, and there is no node graph left
|
||||
to patch afterwards.
|
||||
|
||||
`tools/gltf_fix.py` rewrites the container **before** import: emissive becomes
|
||||
base colour, the unlit flag is dropped. The game shades characters with its own
|
||||
toon material off ALBEDO anyway.
|
||||
|
||||
## Swapping the animation library
|
||||
|
||||
`merge_animations.py` supports two library layouts in
|
||||
`assets/characters/animations/`:
|
||||
`assets/characters/animations/_library.glb` is the CC0 Quaternius Universal
|
||||
Animation Library. Clips map through `LIBRARY_CLIP_MAP` in `tools/retarget.py`
|
||||
(18 mapped: Idle, Walk, Run, Sprint, Jump, Fall, Land, CrouchIdle, CrouchWalk,
|
||||
Dash, Death, Hit, Dance, Grapple, PistolIdle/Shoot/Reload, Throw).
|
||||
|
||||
**A. Multi-clip library GLB (the bundled default).** If `_library.glb` exists,
|
||||
its clips are merged and mapped through `LIBRARY_CLIP_MAP` in
|
||||
`tools/merge_animations.py`. The bundled file is the CC0 Quaternius Universal
|
||||
Animation Library (13 clips: Idle, Walk, Run, Sprint, Jump, Fall, Land,
|
||||
CrouchIdle, CrouchWalk, Dash, Death, Hit, Dance). **Important:** `autorig.py`
|
||||
rigs characters to *this GLB's own skeleton*, so if you replace `_library.glb`
|
||||
you must re-rig characters against the new one (the pipeline does this
|
||||
automatically). To pull in more of Quaternius's 120-clip set, drop in the Pro
|
||||
`_library.glb` and add rows to `LIBRARY_CLIP_MAP`.
|
||||
Characters no longer have to be rigged to the library's skeleton, so **replacing
|
||||
`_library.glb` does not require re-rigging anything** — just rebuild the skins.
|
||||
Missing clips are fine: the game falls back along sensible chains
|
||||
(`Slide → CrouchIdle → Idle`, `WallRun → Run` — see `CLIP_FALLBACKS` in
|
||||
`characters/skinned_player_model.gd`).
|
||||
|
||||
**B. Loose one-clip-per-file (Mixamo).** Delete `_library.glb` and instead add
|
||||
FBX/GLB files named by clip (`idle.fbx` → `Idle`, `crouch_walk.fbx` →
|
||||
`CrouchWalk`). Download from mixamo.com as *FBX Binary, Without Skin, 30 fps*
|
||||
("In Place" variants). In this mode characters must be on the Mixamo skeleton
|
||||
(rig via Mixamo web / AccuRig and pass `--rigged`, or let the heuristic
|
||||
`build_mixamo_armature` fit one).
|
||||
## Measuring, not eyeballing
|
||||
|
||||
Missing clips are fine either way: the game falls back along sensible chains
|
||||
(`Slide → CrouchIdle → Idle`, `WallRun → Run`, etc. — see `CLIP_FALLBACKS` in
|
||||
`characters/skinned_player_model.gd`). A model with just Idle/Walk/Run still
|
||||
animates in every movement state.
|
||||
- `blender --background --python tools/verify_character.py -- <glb>` — the build
|
||||
gate. Every check corresponds to a defect this project actually shipped.
|
||||
- `godot --headless --path . -s res://debug/limb_deform_check.gd -- <glb>` —
|
||||
skins the mesh itself and reports lengthwise stretch and cross-section loss
|
||||
against the skeleton's REST pose, so an unposed model reads exactly 1.00 and a
|
||||
wrong metric is visible immediately. Read its header before trusting a number
|
||||
you add: three earlier versions of this measurement were themselves wrong and
|
||||
nearly caused bad "fixes".
|
||||
- `godot --path . --windowed --resolution 1280x720 -s res://debug/anim_capture.gd -- <out_dir> <skin_id>`
|
||||
— renders every movement state front and side.
|
||||
|
||||
Current Taila, worst over a run/walk/jump/fall/slide/dash sweep: knee
|
||||
cross-section 0.85–0.86, everything else 0.89–1.00, worst stretch 1.16.
|
||||
|
||||
## How it works in-game
|
||||
|
||||
- **`SkinManager` (autoload)** reads `assets/characters/skins/skins.json` at
|
||||
boot. Selected skin persists per-user and is synced to other players via
|
||||
`synced_skin_id`.
|
||||
- **`SkinnedPlayerModel`** loads the GLB at runtime, maps canonical clip
|
||||
names, sets loop modes, blends between clips (0.15 s), and scales
|
||||
locomotion playback speed to actual movement speed.
|
||||
- **First person (owner):** the model renders shadows-only for the owner (the
|
||||
camera sits inside the head, so drawing the mesh would show its inside). It's
|
||||
still fully animated and casts a real shadow; other players see the full body.
|
||||
- **Third person:** press **V** (`toggle_camera_view`) to swap to an
|
||||
over-the-shoulder `SpringArm3D` camera and reveal your own animated model —
|
||||
the easiest way to eyeball that a new skin's animations look right. Firing
|
||||
still uses the first-person camera, so aim is unchanged. Press V again to
|
||||
return to first person.
|
||||
- **Other players** always see the full third-person model, driven by the
|
||||
synced movement state, with the current weapon attached to the hand bone.
|
||||
- **Licensing:** every Sketchfab download writes `<name>.license.json`.
|
||||
CC-BY models require crediting the author — keep these files and surface
|
||||
them in your credits screen before shipping.
|
||||
- **`SkinManager`** (autoload) reads `skins.json` at boot; the selection persists
|
||||
per user and syncs via `synced_skin_id`.
|
||||
- **`SkinnedPlayerModel`** loads the GLB, reads the sidecar, installs
|
||||
`ShooterPoseModifier` (lean / ADS / slide / wall-run / weapon hold) then
|
||||
`SpringBones` — in that order, so the springs react to the FINAL body pose.
|
||||
- **`SkinJointHelper`** runs for every model regardless of rig. It is not a
|
||||
weight repair: linear-blend skinning collapses any joint by cos(θ/2) however
|
||||
good the weights are. It subdivides the knee through helper bones. It resolves
|
||||
the joint's parent from the SKELETON — hardcoding `DEF-thigh.L` meant it
|
||||
silently did nothing on a rig with twist bones, and the knee measured 0.76
|
||||
instead of 0.85.
|
||||
- **First person (owner):** the model renders shadows-only (the camera sits
|
||||
inside the head). Press **V** to swap to the over-the-shoulder camera.
|
||||
- **Licensing:** every Sketchfab download writes `<name>.license.json`. CC-BY
|
||||
models require crediting the author — surface these in a credits screen.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- *Model T-poses in game* — the GLB has no animations; re-run
|
||||
`merge_animations.py` and check it printed `Merged N clips`.
|
||||
- *Mesh deforms badly at shoulders/hips* — heuristic rig didn't fit; rig via
|
||||
Mixamo web or AccuRig and re-run with `--rigged`.
|
||||
- *Character slides while walking* — clips exported with root motion; re-run
|
||||
without `--keep-root-motion` (stripping is the default).
|
||||
- *Skin missing in exported build* — raw `.glb` files must be included in the
|
||||
export: Project → Export → Resources → include filter `*.glb, *.json`.
|
||||
- *Character is a black silhouette* — unlit materials; see **Materials**. Check
|
||||
the built GLB actually has images.
|
||||
- *Model T-poses* — the GLB has no animations, or the retarget produced a frozen
|
||||
rest pose. `verify_character.py` catches both.
|
||||
- *Skirt rides one leg* — a cloth bone got parented to a limb. Cloth must anchor
|
||||
to the trunk only.
|
||||
- *Cloth is rigid* — the sidecar is missing (check the export filter) or the
|
||||
clips are keying cloth bones (`verify_character.py` checks this).
|
||||
- *Cloth flies off the model* — a spring instability. `SpringBones` substeps and
|
||||
clamps for exactly this; do not remove those guards.
|
||||
- *Limbs squash at a stride* — measure with `limb_deform_check.gd` before
|
||||
changing anything. Renders are repeatedly misleading; a slim anime leg at full
|
||||
stride genuinely looks stretched.
|
||||
|
||||
Reference in New Issue
Block a user