About
About Orchestra
Orchestra measures activity on San Francisco streets from street cameras, forecasts it, and grades every saved forecast against what the cameras later measured. The public site publishes only aggregates.
What we are working toward
Our research direction is a future action model: a model of a place, learned from long and persistent observation, that predicts a range of future states with their probabilities and is tested against what actually happens. Fixed cameras watch the same places over time, and every prediction can be checked by the observations that follow it.
The live counts, forecasts and report card on this site are the first measurable pieces of that idea. The forecasts today come from a pretrained time-series model (Chronos-2) with weather, air quality, calendar, daylight and permitted street closures as inputs; they are not yet the learned model of the city we are researching.
Methods
Counting
- Object detection, a segmentation model fine-tuned on machine-generated labels, runs on each camera’s live video. A camera’s count is the average number of people or vehicles visible in its view during one minute.
- A camera is counting when its latest complete minute ended at most 5 minutes before the snapshot. Its value on the Live City page is the mean of at least five valid 1-minute averages in the last 15 minutes.
- Counts are per camera. Views can overlap and see the same person, so city and neighborhood figures are averages per camera, never totals.
- Measured counts are detector estimates, not independently verified ground truth. Detector misses and duplicates are not yet calibrated into the forecasts.
Neighborhoods
- Numbers are published for a neighborhood only when at least 3 of its cameras are counting. Cameras whose location is still being verified count toward city averages only.
- Outlines are DataSF’s Analysis Neighborhoods; Cole Valley, which is not one of them, uses its SF Find Neighborhoods outline. Map positions are the visual centers of those outlines, never camera positions.
Forecasts
- Each camera’s counts are forecast up to 30 days ahead: 15-minute steps for the first week, then hourly. City and neighborhood forecasts average the cameras, each with equal weight.
- Shaded bands are an approximate ±1 standard deviation. They are not calibrated yet, and for skewed counts ±1σ does not promise 68% coverage.
- Every forecast is saved before its interval arrives, and later forecasts cannot rewrite it. Accuracy beyond two days is still being evaluated; month-ahead accuracy has not been validated.
The report card
- Citywide forecasts issued at least 24 hours ahead are compared with later measurements on complete hours, using the same cameras on both sides of each hour.
- Two baselines: the same hour one day earlier, and a seasonal average of the same local hour on at least three earlier days of the same type (workday or weekend), holidays excluded. Each is scored only on hours where it and the forecast both exist, and a baseline may see one more hour of data than the model, which favors the baseline.
- Error reductions carry 90% intervals from a moving-block bootstrap (6-hour blocks, 2,000 resamples, fixed seed). Nothing is scored on fewer than 72 matched hours, and both people and vehicles are always shown, including where a baseline wins.
- Coverage counts the hours that landed inside the band issued with the forecast, never a band calibrated later. A calibrated ±1σ band would hold about 68% of hours.
AI review
- Cosmos 3 Nano, a vision-language model running on Orchestra’s servers, reviews sampled face-blurred scenes in rotation; some signals move a camera forward in the queue. It is not continuous monitoring, and brief events can be missed.
- A possible incident becomes a public alert only when a second review by an external model (OpenAI) agrees. Alerts are machine judgments, not verified by a person. The Live City page shows citywide daily counts of what was flagged for AI review, rejected, expired without a verdict and confirmed by two AI reviews.
- Hourly insights are written by an external model (OpenAI) from counts and forecasts. The public ones are quantitative only, and they may be wrong.
Demographics
The demographics page shows sampled machine estimates over a rolling 7-day window, not census data. No identity or race is inferred.
Cameras, faces and what is published
Original camera recordings are kept privately on our servers for counting and model training. Models on our servers analyze unblurred video, and frames kept privately for them may be unblurred. At the owner's direction, signed-in live video and alert clips preserve faces. Analyzed still images and images sent to outside AI models remain face-blurred, though blur can miss faces. Audio is never decoded. Face-blurred frames and crops go to OpenAI for demographic estimates, incident second reviews and training labels, and at most five per hourly analysis for city insights; alert clips are emailed privately to the owner and two team members. No imagery, tracks or camera positions are published here.
The privacy page lists processors, retention and cookies.
Open data and sources
The aggregates behind these pages are open JSON that needs no key; the developer page documents them. Counts and forecasts are Orchestra’s own. Forecast inputs come from Open-Meteo (weather and CAMS air quality, CC BY 4.0) and SFMTA permitted street closures via DataSF. Neighborhood outlines come from DataSF’s Analysis Neighborhoods and SF Find Neighborhoods, both public domain.
Access
Public pages and open data need no account. Live camera views in the 3D world need an approved Google sign-in and follow the owner's configured live-video mode. OrchestraOS Google accounts sign in directly; other Google accounts can request access by signing in, and the owner reviews each request.
Keys for the versioned API are issued by the operator.
Contact
Email contact is being set up.