The AI pipeline, live

Live video, detection, review and learning across three RTX 5090s and one RTX 3090. GPU percentages are one-minute averages.

  1. Street cameras

    — live

    Their video goes to GPU 1, local frames go to GPU 0, and recordings are saved. Local models analyze unblurred video at the owner’s direction; images sent to outside AI models remain face-blurred.

  2. GPU 1 · Live detection

    YOLO26x finds people and vehicles

    — frames per second

    1m avg —

    It sends hints to GPU 0 about what to look at.

  3. GPU 0 · Live review

    Cosmos 3 Nano screens and reviews

    — routine screens in the last full hour

    1m avg —

  4. Possible incidents

    — flagged · — confirmed

    count unavailable

  5. Saved video

    Recordings and training clips

    Clips from them go to GPU 2.

  6. GPU 2 · Training

    1m avg —

    By day · 5:05 AM–12:45 AM

    V-JEPA 2.1

    —

    — examples per training hour

    At night · 12:45–5:05 AM

    Night jobs

    Cosmos fine-tuning · 1:10

    Detector training · 2:00

    Recordings import · 2:20

    Automatic promotion moves a V-JEPA version into scene reviews on GPU 0 only after every check passes.

    —

  7. Second machine · GPU 0 · RTX 3090

    Cosmos object review

    1m avg —

    Reviews fitting and cached vehicle crops; responses remain machine estimates.

Waiting for one-minute GPU averages…

Other jobs run on the machine’s processors rather than a GPU: incident-model training at 2:05 AM, forecast evaluation at 3:30 AM, saving nearby emergency dispatch calls at 12:35 AM, and an hourly check of possible incidents against public records and responders seen on camera. GPU 2 can also be busy indexing saved video or running a check, with neither lane lit. GPU percentages average the last minute’s recorded samples. Each GPU’s role, on the updates page

AI · recorded training metrics

AI training losses

What the losses measure, and whether scores improve on examples kept out of training.

All GPUs · 1-minute averages

Three RTX 5090s on conductor, plus one RTX 3090 on conductor-dev; useful queued work uses spare capacity while live inference, memory headroom and cooling take priority.

Waiting for current GPU measurements…

GPU processes

Checking operator access…

Open the secure process view

Recorded model losses

Loss measures disagreement with a reference answer or label. Lower means a closer fit to that reference. Each chart says what is being compared.

Does the model improve? The plots after the losses show scores on examples kept out of weight updates. Higher is better on those scores; reference labels can still be wrong.

Training trends. The line is a trailing mean; shading shows ±1 local standard deviation. Choose a smoothing window or Raw below each training chart.

Technical details · loss, validation and restarts

The optimizer changes trainable weights to reduce an objective, usually a mean over a batch or an update window. Answer-token cross-entropy penalizes assigning low probability to the reference answer tokens. Detection losses separately penalize box, class and mask mismatches. Machine-written reference labels can be wrong.

Training examples produce gradient updates. Validation examples measure the current model without updating it in that evaluation; test examples provide a separate held-out comparison. Repeated validation inspection can still influence selection, so test results and real error cases matter.

A cumulative process mean averages a task’s losses since that trainer process started. It can jump when a new process starts its average, even if training resumes the same model weights and global step. Process segments stay separate; a denser line does not create more independent evaluations.

A lower loss is not accuracy and does not prove a better model in use. Compare the same task, split, label source and run segment, then check held-out quality and examples of errors.

Loss history is loading. No values are assumed.

Loading the recorded model state…

Video & interactive demos

Inspect motion and evidence over time. Real archive replay uses the existing recording access controls; browser simulations are labeled synthetic.

Loading the demo collection…

Open the interactive demo gallery

Recorded metrics

Each model keeps its native metrics. Choose a model, then a recorded run in each chart. Training, validation and test have separate charts, each with its own scale.

Open public data

Every model stays listed, including models with no loss samples or no local training. Record timestamps describe training records, independently of the GPU sample time. “Last reported” marks model records older than 15 minutes. Charts show all published raw points for a selected run. A lower loss alone does not establish better live performance or promotion into use.

Connecting to live telemetry…

Waiting for a response

GPU samples

GPU telemetry for both machines; each row has its own sample time.
Host / GPUWorkloadStateUtilization · 1m meanMemoryTemperaturePowerSample
conductor · GPU 0RTX 5090—Waiting—Sample mean unavailable— / — MiB——No sample reported
conductor · GPU 1RTX 5090—Waiting—Sample mean unavailable— / — MiB——No sample reported
conductor · GPU 2RTX 5090—Waiting—Sample mean unavailable— / — MiB——No sample reported
Second machine · GPU 0RTX 3090—Waiting—Sample mean unavailable— / — MiB——No sample reported

Utilization is the average of recorded samples in the last minute. Refreshes every 5 seconds while this page is visible. Samples older than two minutes are marked stale and their current readings are unavailable. A dash means no valid measurement was reported; a measured zero is shown as 0. Memory is used / total, in MiB. Scroll the table to see every column on a small screen.

What each model does live

Live means right now, on the cameras streaming to our GPU machine.

ConnectingOur GPU machine · loading live data

Checking which versions are live…

Explore the synthetic archive demo Generated examples · no real footage or findings

Live street cameras
—
Loading…
Routine camera screens, last full hour
—
Loading…
V-JEPA training examples today
—
Loading…
Cosmos fine-tuning runs completed
—
Loading…

GPU 0 · in live use

Cosmos 3 Nano reviews the cameras

A vision-language model from NVIDIA’s Cosmos family: it looks at an image and answers in words. Here it asks of each live camera’s latest frame whether it shows a possible incident, such as fire or smoke, a collision, a person down, a fight or a vehicle break-in. Every few minutes each camera also gets a fuller scene review. A possible incident becomes an alert only after an external AI model agrees on a second review. Alerts are not checked by a person.

Routine camera screens, last full hour
—Loading…
Median time since each live camera was last screened
—Loading…
Routine scene reviews, last full hour
—Loading…
Possible incidents
—Loading…

Confirmed means Cosmos 3 Nano and an external AI model both agreed. Neither is a person’s check.

Version: loading…

Fine-tuned Cosmos runs are kept for testing; none is used live.

GPU 2 · training

A motion link from V-JEPA 2.1 into Cosmos 3 Nano

V-JEPA 2.1 is a video model from Meta that learned from video itself, without labels. We are training a link from it into Cosmos 3 Nano so scene reviews can take in the last 8 seconds of motion, not just one frame. The link trains on clips from our live cameras and saved recordings, using puzzles whose answers we know because we made them.

Status
—Loading…
In live use
—Loading…
Go-live checks run
—Loading…
Automatic promotion
—Loading…

Automatic promotion moves a version into scene reviews only after it passes 13 automatic checks on test cameras it never trained on and a canary of up to 3 hours on GPU 0. Its first 24 hours in use are a probation: any failure rolls it back. An operator can also put a version in use or roll one back by hand; the In live use row says when that happened.

Training now

GPU 2 trains both models: V-JEPA 2.1 from 5:05 AM to 12:45 AM, and Cosmos 3 Nano fine-tuning at 1:10 AM and after a restart of our server. Every figure here is written by the training jobs themselves. Times are Pacific.

V-JEPA 2.1: motion puzzles, round by round

Loading…

Loading the trainer’s status…

—

Each step trains on 32 examples. Each round then ends with a test on validation cameras, except in qualification.

Examples per training hour
—
Loading…
Examples trained today
—
Loading…
Training-step hours today
—
Loading…
From the last 24 hours
—
Loading…

Loading the latest validation results…

  • Accuracy minus chance, by round
  • 95% range
  • Chance
  • Always giving the set’s most common answer

These scores are on validation cameras. To go live, a version must score more than 5 points above chance on each puzzle on test cameras, with a significance test, as one of its automatic checks. Each chart shows the newest rounds that fit its width; Show the numbers lists every round.

Arrow of time

Loading…

Is the clip playing forward or backward?

—chance 50% · 2 possible answers

Loading…

Loading the training history…

Order

Loading…

The clip’s three parts may be shuffled. Which order is right?

—chance 16.7% · 6 possible orders

Loading…

Loading the training history…

Span

Loading…

Does the clip cover 8, 16 or 32 seconds of real time?

—chance 33.3% · 3 possible answers

Loading…

Loading the training history…

Sync

Loading…

Was this still taken within a second after the clip ends, or 4 to 12 seconds later?

—chance 50% · 2 possible answers

Loading…

Loading the training history…

Loss against machine labels

—

Loading the training history…

Answer-token loss against labels written by an external AI model, on validation cameras. Lower means closer to those labels; it is not accuracy, and the labels can be wrong.

Loading…

Cosmos 3 Nano: fine-tuning runs

Loading…

Loading the job’s status…

Runs start at 1:10 AM and once shortly after each restart of our server, when the labelled set has changed. Each trains a small adapter on a 4-bit copy of the model, starting again from the released weights, so runs do not build on each other.

Loss on held-out examples, before and after each run; lower means its answers are closer to the machine labels. before after

  1. Loading the run history…
Labelled examples
—
Loading…
Labelled in the past day
—
Loading…
Incidents in the labels
—
Loading…

A lower held-out loss means a run’s answers moved closer to the machine labels on those held-out examples; it does not show better reviews. Labels come from an external AI model (from OpenAI) looking at face-blurred frames, and no person checks them; no camera appears in more than one of the training, validation and test sets. No run is in live use: nothing loads one live.

What these numbers mean

And what they are not.

Camera screens
One short question to Cosmos 3 Nano about one camera frame: does it show a possible incident? Routine screens take the cameras in turn; extra screens of cameras that something drew attention to are not counted here. Brief events between screens can be missed. It is not a count of incidents.
Median time since a camera’s last screen
Half of the live cameras with a screen on record were last screened at most this long ago. A screen stays on record for 14 days; live cameras without one are left out and counted separately.
Routine scene reviews
Fuller descriptions of a camera’s scene, made in rotation, in the last complete hour.
Possible incidents
Flagged: a screen, a detector hint or a review proposed one. Confirmed: Cosmos 3 Nano and an external AI model both agreed. Neither is checked by a person. Counting starts again after a gap in the server’s records, so “since” can be later than seven days ago.
Motion puzzles
Questions about real footage from our cameras that we reversed, shuffled, sampled more sparsely or paired with a still. Because we made the change, the right answer is known exactly. They test whether motion and time reach Cosmos 3 Nano, not whether it recognizes events. They are not incident accuracy.
Chance
What random guessing among the allowed answers scores on average: 50% for arrow of time (2 answers), 16.7% for order (6 orders), 33.3% for span (3 lengths) and 50% for sync (2 answers). These come from the training code. A set of 100 clips can lean toward one answer; when we know that lean, we also show what always giving the most common answer would score.
95% range
The spread expected from the clips scored alone. When it reaches down to chance, or to what always giving the most common answer would score, the result cannot be told apart from guessing. It does not adjust for looking at every round.
Validation and test cameras
Validation cameras never train the model and are used to follow its progress. Test cameras, a separate set, are read only by the go-live checks. Both are held-out camera views, not new places: some look at streets that training cameras also see.
Loss against machine labels
Answer-token loss: how unlikely the model found the label’s words, averaged per word piece. Lower means its answers are closer to labels written by an external AI model (from OpenAI). It is not accuracy, and machine labels can be wrong.
Examples per training hour
Examples divided by the time spent in training steps since the trainer last started. Time spent testing and saving is not counted, so it runs faster than examples per clock hour.
Examples today
Counted by the trainer when each round ends or it stops, since midnight Pacific time.
From the last 24 hours
Share of a round’s examples cut from video captured in the past day. A round does not start below 25%.
Rounds and stages
A round is 250 steps of 32 examples, then a test on validation cameras. Stage 1a trains only the bridge that turns V-JEPA 2.1’s reading of an 8-second clip into 96 tokens for Cosmos 3 Nano; Stage 1b also trains a small adapter inside Cosmos. In both, V-JEPA 2.1’s own network stays frozen: what learns is the bridge and the adapter. A qualification stage then trains two copies side by side, one that also tunes the top blocks of V-JEPA 2.1’s own network and one that keeps it frozen, to check whether tuning helps; its rounds skip the validation test. Stage 2 continues with the copy qualification picked, and its rounds can become candidates for live use.
Why not hours of video
Each V-JEPA example is one 8-second clip (up to 32 seconds for span), clips are reused across examples, and old clips are removed over time, so hours of video would overstate what the model has seen. Cosmos fine-tuning learns from labelled frames, not video.
Live use
Automatic promotion moves a V-JEPA version into scene reviews only after 13 automatic checks on test cameras and a canary of up to 3 hours on GPU 0, and its first 24 hours in use are a probation that rolls it back on any failure. An operator can also put a version in use by hand, which the checks do not gate, or roll one back; the page says so when that is how the version in use got there. Until a version is in use, live reviews use the released Cosmos 3 Nano. Cosmos fine-tuned runs are kept, but nothing loads them live or promotes them automatically.
What we do not publish
No images, video, camera IDs, locations or tracks; no list of held-out cameras; no names or addresses; only totals. At the owner's direction, signed-in live video and alert clips preserve faces. Models on our machine analyze unblurred video: the people-and-vehicle detector does, and the motion model trains partly on unblurred clips from saved recordings. Analyzed stills and images sent to outside AI models remain face-blurred, and blur can miss faces.

These numbers come from /api/public/models, /api/public/machine and /api/public/compute, open to anyone without a key. See the developer page