Domain randomization and curricula¶
Many knobs of a generator are dynamic fields: per-env
[E] arrays the stages read at every generation, so each env can run its own value and
the values can change between generations with no rebuild and no new capture.
Two tools write them:
randomize (domain randomization):
randomize=adds a stage that draws a field per env, inside the generation, from a range you set (Per-env draws: randomize=);a curriculum: you write the live arrays yourself, from the host or from a kernel in your own graph (Curricula: writing the live arrays).
This page lists the dynamic fields, then covers both tools, what each env was built
with, the difficulty axes, the course size and the utilities it affects, and a
Course curriculum. Static fields (counts, buffer sizes, the road width) need a new
generator: Parameters.
Dynamic fields¶
A dynamic field is a per-env [E] array (float32, or int32 for a count) the
kernels read at every generation. You can draw a float32 one per env with
randomize=, or write any of them yourself. The dynamic fields of the built-in stages
are:
the built-in generators’ shape knobs:
radandhandle_clamp_frac(bezier),hull_displacement(hull),polar_radial_jitterandpolar_angular_jitter(polar),voronoi_radial_variationandvoronoi_angular_jitter(voronoi),checkpoint_radius_min_frac,checkpoint_angle_jitterandcheckpoint_turn_rate(checkpoint). The polar, voronoi and checkpoint gate generators have the same ring knobs (notcheckpoint_turn_rate: they do not steer). Each generator page gives their bounds, effective range and measured yield (Per-env shape knobs);the course size (
size,[0.1, 10]) of every built-in generator butexternal:generator.sizeon tracks and on thebezier/hullgate generators,normalize.size(thenormalize.extentstage) for thepolar,voronoiandcheckpointgate generators (Size: the sizing rule);the corner count of
bezierandhull:min_num_points, anint32in[2, max_num_points]that you write (setor your kernel);randomize=does not drawint32fields (Corner count);the gate bar width,
gate_frames.default’sgate_width(keygate_frames.gate_width,>= 0; Per-env bar width and course size);the elevation profiles’
z_minandz_max(uniform,random_walk,noise),z_max_step(random_walk) andz_noise_amplitude(noise), on tracks, on gates and on the centerline before the stations alike (Elevation knobs);width.from_channel’sscale,loandhi, and every field a plugin declares withplugin.dynamic(Runtime parameters and inputs).
Everything else is static (max_num_points, knot, anchor and checkpoint counts,
half_width, spacing, n_max, edgy, the core scale, z_base, the
relax and validity tolerances, the repulsive growth knobs, the gate generators’
min_num_points): build a new generator to change it. gen.params lists the stages
that have a dynamic field; registry.describe(kind, name).params_schema marks each
field ("x-dynamic": true).
Per-env draws: randomize=¶
The preset argument presets.track(randomize={"<stage id>.<field>": (lo, hi)})
(and presets.gates) draws each env’s value uniformly in [lo, hi) at every generation. The generator’s
stage id is "generator", the elevation’s "elevation", in presets.track and
presets.gates alike:
from track_gen import TrackGenerator
from track_gen.pipeline import presets
E, device = 64, "cuda:0" # or "cpu"
spec = presets.track(
num_envs=E, device=device, generator="polar",
elevation=("noise", {"z_min": -0.3, "z_max": 0.3, "z_noise_amplitude": 0.1}),
randomize={"generator.polar_radial_jitter": (0.3, 0.85),
"elevation.z_noise_amplitude": (0.0, 0.3)},
)
spec.ids
# ('randomize.generator.polar_radial_jitter', 'generator', 'resample', ...,
# 'randomize.elevation.z_noise_amplitude', 'elevation', 'validity', 'lift')
gen = TrackGenerator(spec)
gen.params["generator"]["polar_radial_jitter"] # [E]: what each env was drawn
gen.generate(advance=True) # new tracks, new draws
The draw. Env
egetslo[e] + u * (hi[e] - lo[e]), withuuniform in[0, 1)from a stream keyed on the env’s effective seed (Seeds, epochs and the generation counter) and the randomizer, clamped to the field’s bounds. The draws are deterministic per(seed, epoch)(epoch), differ across envs and across randomizers, and change withadvance=True(each replay of a graph that recordsgenerate(advance=True)redraws) or a reseed. A masked generation redraws the masked envs only; the others keep their values and their rows.Placement. Each entry adds a
randomize.uniformstage, with the idrandomize.<stage>.<field>, right before its target, so the target reads this generation’s draw. In a spec you edit by hand,spec.with_randomize({...})inserts them the same way; a spec built from a config takes them the same way (legacy.to_pipeline_spec(cfg).with_randomize({...}); the configs have norandomizefield).What can be drawn. A dynamic
float32field. A static,int32or unknown field raisesPlanErrorwhen the generator is built, listing the fields of the pipeline that can be drawn (with a hint where another key holds the axis:generator.sizeon apolargate pipeline namesnormalize.size). So do a randomizer listed after its target, two randomizers on one field, an initial range outside the field’s bounds, andz_min/z_maxrandomizers that can cross (Elevation knobs). Per-env randomization: randomize= has every rule.A drawn field belongs to its randomizer.
gen.params[stage][field]holds the current draws (read it to log what each env got), butsetraisesValueErrorand a direct write is overwritten at the next generation. To move the distribution, write the randomizer’s own dynamicloandhi:gen.params["randomize.generator.polar_radial_jitter"].set("hi", 0.6)(a curriculum on a distribution). To change it in a spec,spec.replace(...)that randomizer;with_randomizerefuses a field the spec already randomizes.Cost. Two small launches per randomizer per generation; none for a pipeline without one.
The built-in dynamic knobs¶
stage id: stage |
dynamic fields (bounds) |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The elevation stages are the same classes on a track, on gates
(Elevation on gates) and on the centerline before the stations
(3D station spacing), so the same keys ("elevation.z_max") work in all
three. external has no dynamic field: it ingests your geometry as given. bezier’s
deprecated style_sampling draws rad / handle_clamp_frac / its scale itself
when their ranges are set; such a field cannot also take a randomizer or a set. Use
randomize= instead (Style sampling (deprecated) and its migration, style_sampling).
Curricula: writing the live arrays¶
gen.params[stage][field] is the live [E] array of a dynamic field, at a stable
address. Write it with gen.params[stage].set(field, value) (a scalar for every env,
or an [E] array of the field’s dtype), or from a kernel recorded in your own graph
before the generation: the next generation reads it, for the envs it regenerates. A
curriculum kernel reads your training signal and writes the difficulty, inside the graph
that also regenerates the tracks:
import warp as wp
from track_gen import TrackGenerator
from track_gen.pipeline import presets
E, device = 64, "cuda:0" # or "cpu"
@wp.kernel
def difficulty_k(success: wp.array(dtype=wp.float32),
jitter: wp.array(dtype=wp.float32), z_max: wp.array(dtype=wp.float32)):
e = wp.tid()
level = wp.clamp(success[e], 0.0, 1.0)
jitter[e] = 0.3 + 0.55 * level # rounder tracks first, then eccentric ones
z_max[e] = 0.1 + 0.4 * level # then bigger hills
spec = presets.track(
num_envs=E, device=device, generator="polar",
elevation=("noise", {"z_min": -0.3, "z_max": 0.3, "z_noise_amplitude": 0.1}),
randomize={"elevation.z_noise_amplitude": (0.0, 0.3)}, # drawn: not written below
)
gen = TrackGenerator(spec)
jitter = gen.params["generator"]["polar_radial_jitter"] # [E] float32, stable
z_max = gen.params["elevation"]["z_max"]
success = wp.zeros(E, dtype=wp.float32, device=device) # your training signal
with wp.ScopedCapture(device=device) as capture:
wp.launch(difficulty_k, dim=E, inputs=[success, jitter, z_max])
gen.generate(advance=True)
success.fill_(1.0)
wp.capture_launch(capture.graph) # each replay reads the new values
The same works from the host between generations
(gen.params["generator"].set("polar_radial_jitter", 0.6)), and on cuda in the
generator’s own CUDA graph or in yours (Rules for recording into your graph). A field you write must not
be drawn by a randomizer (above): this generator draws only the hill amplitude and leaves
polar_radial_jitter and z_max to the curriculum kernel.
One rule for bounds. A dynamic built-in knob declares its bounds (The built-in
dynamic knobs). A scalar is checked on the host: a value outside the bounds raises
ValueError, at construction, from set, and for a randomizer’s initial range
(PlanError). The comparison is made in float32, the precision the array holds, so
a value read back from an array is always accepted. An array value (an [E]
set, your own kernel, a randomizer’s draw from bounds you wrote) is not checked: the
kernel clamps it to the bounds when it reads it (a NaN passes through, and an unbounded
side, such as z_min’s or handle_clamp_frac’s upper one, is not clamped). So a
curriculum may overshoot a bound without failing; the excess piles onto the bound.
gen.params[stage].bounds(field) returns the (lo, hi) a value is held to (None
for an open side), narrowed per generator where the static configuration narrows it:
min_num_points’ are (2, max_num_points), and set refuses a scalar outside
them (and a float: it is an int32).
Check the yield at both ends of every range. Many knobs lose yield, or stop
mattering, well inside their bounds: rad has no effect above handle_clamp_frac
(drawing both independently leaves rad inert in every env where it lands at or above
the clamp: keep hi(rad) <= lo(handle_clamp_frac); a generator.rad randomizer
whose hi exceeds every value the clamp can take emits a UserWarning when the
generator is built), voronoi_angular_jitter is legal up to pi but its yield is at
most 0.85 from 0.5 (0.71-0.85 by layout) and near zero from 1.2,
checkpoint_radius_min_frac at 0.99 keeps two thirds of the tracks. Measure the
invalid env fraction at each end (Invalid envs).
With a stage that runs on the host (a plugin generator or relax stage), the new values are read at the start of the generation’s first recordable part: a write must come before it to apply to that generation (Capture classes and segments).
What each env was built with: in_effect and bounds¶
At the start of a generation the generator copies the live arrays into the working
copies its kernels read, for the envs it regenerates only (how). Under
generate(advance=True, mask=m) an env left out keeps the value its current track was
built with, so the live array may already hold a newer value than that env’s track.
gen.params[stage].in_effect(field) answers “which difficulty is each env running?”:
the [E] values the stage read for each env’s current rows (the drawn values, for
a field a randomizer draws):
import warp as wp
from track_gen import TrackGenerator
from track_gen.pipeline import presets
E, device = 64, "cuda:0" # or "cpu"
gen = TrackGenerator(presets.track(num_envs=E, device=device))
done = wp.array([1] + [0] * (E - 1), dtype=wp.int32, device=device) # your done mask
my_log = wp.zeros(E, dtype=wp.float32, device=device)
p = gen.params["generator"]
p.set("size", 1.5) # the curriculum's next level
gen.generate(advance=True, mask=done) # only the done envs pick it up
built = p.in_effect("size") # [E]: what each env's current track read
p.in_effect("size", out=my_log) # or: one copy into your own [E] array
p.bounds("size") # (0.1, 10.0): what a built-in clamps to
A view, or a copy. Without
out=it returns the internal array: no copy, a stable pointer, usable in your graph. Read it, never write it: a write (an in-place torch op onwp.to_torch(...)included) changes what the unmasked envs report, and what a randomizer redraws for them. Without=(a contiguous[E]array of the field’s dtype on the generator’s device;ValueErrorotherwise) it enqueues one copy into it, on the streamsetwould use (stream=, the ambient scope, the generator’s stream), recordable in your graph, and returnsout.KeyErrorfor a static or unknown field.Values read, values used. An array value is not range-checked, and the built-in generator, elevation,
normalizeandgate_framesstages clamp it tobounds(field)where they read it. So the value a built-in kernel used isclamp(in_effect(f), *bounds(f)): equal toin_effect(f)for every in-range array, every scalarset(checked on the host) and every randomizer draw (clamped when drawn). Write in-range values, or book the clamp. Exceptions, used as written:width.from_channel’sscale/lo/hi(it clamps its output), a randomizer’slo/hi(it clamps its draw),style_sampling’s draws, and, in aCoursein gates mode,gate_frames.gate_width(its floor> 0is checked byseton a scalar only: an array value<= 0is used as0).When it is current. The arrays are written on the generation’s stream: read them on that stream (a later launch or node of the same graph,
out=there) or after waiting for the generation. A drawn field behind a stage that runs on the host is current once that part of the generation has run (until then the array holds the previous generation’s draws). Between asetand the next generation,in_effectreturns the old values; before any generation, the initial value.
A drawn field is redrawn for every env, but an env left out of a masked generation keeps
its effective seed and its lo / hi, so it redraws the same bits: in_effect
of a drawn field is the draw each env’s track was built with too.
What stays static, and how to vary it anyway. Mix generators or configurations in
one batch (Mixed batches), each writing its own env range of one
Track, or write a plugin generator whose knobs are plugin.dynamic (with declared
lo / hi, Runtime parameters and inputs).
Difficulty axes and the knobs that reach them¶
Axis |
|
|---|---|
Curviness, corner sharpness |
|
Irregularity |
|
Eccentricity, inlets |
|
Hills |
|
Grade |
|
Course size |
|
Corner count |
|
Corner asymmetry |
not reachable per env: |
Gate bar width |
|
Gate count |
not reachable per env: a gate generator’s |
The gate pipelines take the same keys for the polar, voronoi and checkpoint ring knobs
and the elevation (presets.gates(randomize=...)).
Elevation knobs¶
The window [z_min, z_max] clips every profile, so the knob that sets the difficulty
depends on the profile (Per-env altitude: randomization and curricula):
Profile |
Difficulty knob |
Window |
|---|---|---|
|
|
keep |
|
|
a wide window: at |
|
|
a wide window symmetric around |
When the generator is built, an inert window emits a UserWarning: z_min == z_max
with neither drawn, while z_max_step (random_walk) or z_noise_amplitude
(noise) can be positive (its configured value, or the hi of its randomizer). A
window drawn by randomizers is open and does not warn; a window you open only at run time
(set) cannot be seen then, so a curriculum starting flat gets the warning.
Randomizers on z_min and z_max that could cross (hi of z_min above lo
of z_max, or one drawn range crossing the other’s fixed value) raise PlanError
when the generator is built. A crossing you write yourself at run time is not refused:
uniform then draws in [z_max, z_min], random_walk and noise return
z_max everywhere.
Size: the sizing rule¶
size is a per-env factor on the core scale, applied where each generator applies
scale: the corner grid of bezier and hull (and their gate generators), the
normalization target of polar, voronoi and checkpoint (on gates, the
normalize.extent stage), the output of repulsive (whose scale sets its
physics domain instead). At size = 1 every output is byte-identical to a pipeline
without it; bezier / hull at scale = 1 and size = s are bitwise scale =
s, the normalized generators within one float32 rounding of it. external has
no size: it ingests your geometry as given (a size would move a circuit ingested in
world coordinates about the data’s origin); scale centerline.xy in your own graph.
Initial value:
presets.track(size=)/presets.gates(size=),TrackGenConfig.size/GateGenConfig.size; per env:randomize={"generator.size": (lo, hi)}(ornormalize.size),gen.params["generator"].set("size", ...), your kernel. It is not the batch size (num_envs).A route that cannot apply it raises
ValueErrorinstead of ignoring it: a flatsize != 1with a generator that declares nosize(external, a plugin whose Params lack it, a legacyGeneratorSpec),presets.track(generator="external", size=2.0),size=together withgenerator_params={"size": ...}, and the frozen legacy module functions (generate_*_warp) given a config withsize != 1.
Plan view only. size scales xy. The widths (half_width, spacing, the relax
bands, gate_radius, gate_width) and z (z_min / z_max, the noise
amplitude, z_valid_grade) are absolute: size is the “course larger relative to
the car” axis. So:
under constant spacing a larger course has more points (the rule below), and small sizes lose yield to the absolute road width (self-overlap after the offset): for
bezier, 0.945 valid atsize = 0.5;elevation.uniformplaces its knots at arc fractions andnoiseruns its harmonics in normalized arc length, so their grades fall as~1 / size(uniformwithz_max = 0.3: median per-track maximum grade 0.666 / 0.337 / 0.225 at size 1 / 2 / 3): scalez_max/z_noise_amplitudewith the size to keep the grade.random_walk’sz_max_stepis a grade cap per unit arc and does not depend on the size. On gates the spacing grows with the size, so thez_valid_gradecheck passes more envs. A joint size and hills curriculum is therefore not orthogonal.
Truncation. Size changes no buffer and no launch. The constant-spacing resample emits
floor(size * L / spacing) + 1 points (L the raw perimeter at size 1) into
n_max slots; a track that needs more is flagged in Track.truncated and fails
validity (Invalid envs). A gate pipeline with source="relaxed_stations" fails
the same way (its centerline is resampled into n_max); anchor-sourced gates have a
fixed count and are not affected (their spacing grows). At the defaults (n_max = 384,
half_width = 0.1, spacing 0.06; CPU), the longest loop (rounded up) and its p99
over 8192 envs (seeds 0..4095 and 7919 * e + 13 for e < 4096) and the yields
over E = 256 (seeds 7919 * e + 13); repulsive: both over E = 32, seeds
7919 * e + 13:
generator |
longest raw loop at size 1 (p99) |
|
valid at size 0.5 / 1 / 2 / 3 |
|---|---|---|---|
|
8.68 (7.17) |
~2.6 |
0.945 / 1.000 / 0.996 / 0.996 |
|
8.30 (6.90) |
~2.8 |
0.828 / 0.992 / 1.000 / 1.000 |
|
6.24 (5.67) |
~3.7 |
0.961 / 1.000 / 1.000 / 1.000 |
|
5.01 (4.78) |
~4.6 |
0.996 / 1.000 / 1.000 / 1.000 |
|
7.26 (6.18) |
~3.2 |
0.902 / 0.992 / 0.996 / 0.992 |
|
14.48 (bound 14.83) |
~1.6 |
— / 1.000 / 0.000 (every env truncated) / — |
The sizing rule, for a size range up to size_hi (L_max from the table, times
scale, or measured on your configuration with gen.program.tap("raw.xy")):
n_max >= size_hi * L_max / spacing + 1(else truncation, an invalid env);num_points >= size_hi * L_max / spacing: the raw loop keeps its staticnum_points, so its point spacing issize * L / num_points(bezier, 256 points: median per-env largest raw segment 0.019 / 0.038 / 0.057 at size 1 / 2 / 3, against the output spacing 0.06); a coarser raw loop makes the constant-spacing resample interpolate a polyline;the utilities’ capacities and resolutions (Size and the utilities).
import warnings
from track_gen import TrackGenerator
from track_gen.pipeline import presets
E, device = 64, "cuda:0" # or "cpu"
with warnings.catch_warnings(record=True) as caught:
warnings.simplefilter("always")
TrackGenerator(presets.track(num_envs=E, device=device,
randomize={"generator.size": (1.0, 3.0)}))
print(caught[0].message) # generator.size=3.0 truncates at n_max=384 (...): raise n_max to >= 435 ...
gen = TrackGenerator(presets.track(num_envs=E, device=device, n_max=448,
randomize={"generator.size": (1.0, 3.0)}))
Build-time warning. When the generator is built, a generator.size randomizer whose
hi (or a construction value) would truncate the generator’s nominal longest loop
emits one UserWarning naming the n_max that fits. The nominal loop is exact for
repulsive (from its parameters); for the others it is the longest loop of the table
(so the warning fires from the “truncates from” column), scaled by the knobs that
lengthen it: bezier / hull’s max_num_points ((max_num_points / 13) **
0.6 above 13), hull_displacement (its initial value or the hi of its
randomizer: 1 + 0.6 * (d - 0.15) above 0.15) and style_sampling’s
scale_range. It is not checked when polar’s polar_num_knots /
polar_radial_jitter, voronoi’s voronoi_num_sites /
voronoi_control_points / voronoi_angular_jitter, or checkpoint’s
checkpoint_count / checkpoint_radius_min_frac / checkpoint_angle_jitter /
checkpoint_steer_gain / checkpoint_lookahead_frac / checkpoint_best_of_k
differ from their defaults or are drawn (the other knobs keep the check), nor for a
plugin generator. The longest loop is the maximum of 8192 envs: a larger batch can have
a longer one, so a few of its envs can truncate just below the threshold. It is a
heuristic, so a warning, never an error, and a curriculum written with set or from
your graph is not checked: size n_max for its largest value yourself (measure
L_max with gen.program.tap("raw.xy")).
Size and the utilities¶
The runtime utilities size some buffers and grids once. A size curriculum that starts small and grows must not outrun them while the tracks stay valid:
Checkpoint and prop rings.
Courseand the componentstools.Checkpoints/tools.Propssize their rings from the longest track the generator can produce, with no readback (Checkpoint capacity):max_checkpoints = max(3, ceil(max_loop_length / checkpoint_spacing)) max_props = max(3, ceil(max_loop_length / prop_spacing)) max_loop_length = 1.5 * n_max * spacing + dZ
A track that fits
n_maxhas a plan-view perimeter belown_max * spacing; the1.5covers the relax growth and a boundary’s extra~2 pi * half_width(measured: at most 0.90 ofn_max * spacingjust below truncation), anddZthe elevation’s climbs, from the declared ranges of the elevation knobs (0when flat). At the defaults that is 70 checkpoints at spacing 0.5, at every size. A generator with no declared elevation range (z_min/z_maxdeclare none;z_valid_grade > 0gives one), with a variable width, one that provides its own altitudes (no grade) or half-widths, or a plugin geometry stage declares no bound: thereCoursereads its construction batch back, and the components need an explicit capacity. The legacyCheckpointSampler(track, spacing)/PropSampler(track, spacing)withmax_* = Nonealso read the batch back (ceil(1.5 * longest valid perimeter / spacing)over the bound batch). A readback taken at size 1 clips the grown tracks’ rings (truncated), and progress and lap rewards read them (bezier,E = 32, seeds7919 * e + 13, checkpoint spacing 0.5, 20 checkpoints derived at size 1: truncated on 0 / 0 / 44 / 94 % of the valid envs at size 1 / 1.5 / 2 / 2.5): pass the capacity explicitly there, or construct atsize = size_hiandsetthe size down.SDF and heightfield grids. The
sdfcollision backend and the heightfield baker cover each env’s bounding box with a fixedR x Rgrid, so the cells grow with the size while the road stays absolute (bezier,R = 128: 12.3 / 6.7 / 4.5 cells across the road at size 1 / 2 / 3; OOB flags agree withsegmentson 99.93 / 99.80 / 99.68 % of the boxes). Collision accuracy then varies per env with the difficulty. Rule:sdf_resolution/heightfield_resolution >= R_ref * size_hi,R_refthe resolution accurate enough at size 1; memory grows assize_hi^2. Or use the exactsegmentsbackend (Collision backends). The componentstools.BandCollision(method="sdf", size_hi=...)andtools.HeightFieldBake(..., size_hi=...)warn atattachwhen the rule is broken (Grid resolution and course size). Their defaultsize_hi="auto"reads the generator’s declared range (agenerator.sizerandomizer’s constructionhi, else the constructionsize), so aCoursebuilt withCourse.trackorCourse(generator, components)warns (Composing a course);Course(CourseConfig)declares nosize_hiand does not warn.
A course curriculum¶
A course drives its generator’s dynamic fields like any generator:
course.generator.params[...]. Build the course from a spec with the randomizers or
the initial values you want, then move them between episodes, before
course.regenerate(...):
from track_gen import tools
from track_gen.course import Course
from track_gen.pipeline import presets
E, device = 64, "cuda:0" # or "cpu"
spec = presets.track(num_envs=E, device=device,
randomize={"generator.size": (1.0, 2.0)})
course = Course.track(spec, checkpoint_spacing=0.5, seeds=0,
collision=tools.BandCollision("sdf", sdf_resolution=256))
# later, as the agents improve: widen the size range for the next regenerations
course.generator.params["randomize.generator.size"].set("hi", 2.5)
The course is ready at construction, at the spec’s sizes. At flat elevation the default
checkpoint capacity already holds every track that is not truncated, at any size
(Size and the utilities). The SDF and height-field grids do depend on the size:
BandCollision and HeightFieldBake default to size_hi="auto", the largest
size of the spec (a randomizer’s construction hi on generator.size, else the
generator’s construction size), and warn at attach when the resolution does not
cover it (ceil(128 * size_hi): 256 above; Grid resolution and course size). The
check reads the construction range only: a curriculum that widens it later, as above, is
not checked again, so pass the curriculum’s final hi explicitly
(tools.BandCollision("sdf", sdf_resolution=320, size_hi=2.5)), or use
collision="segments". With elevation and no declared z range, the checkpoint ring is
sized from the construction batch: pass max_checkpoints, or construct at the largest
size and set it down.
A Course(CourseConfig) course declares no size_hi and never warns; its
generator’s size is the config’s TrackGenConfig.size. The bar width of a gates
course is course.generator.params["gate_frames"] (Per-env bar width and course size). To
end an episode on a new course at the new difficulty, regenerate the done envs
(RL auto-reset); in_effect tells which difficulty each env is running.