Start with a testable question.
“Does it detect fire?” is too broad to evaluate meaningfully. Define the event you care about, the camera view, the environment, and the response you expect from the site team.
- Target hazard and a clear reference point for when the event begins.
- Camera model, firmware, stream settings, mounting, and viewing distance.
- Ordinary activity, expected heat sources, and environmental variation.
- Existing detection baseline and the action a new alert would trigger.
Pre-ignition warning and post-ignition detection are different outcomes. Each needs its own sensing assumptions, reference events, and evaluation criteria.
Measure the things an operator needs to know.
One accuracy percentage cannot describe a monitoring system. Agree the metrics, observation period, and acceptance thresholds before final evaluation.
| Measurement | What it tells you |
|---|---|
| Detected and missed events | Which independently labelled target events produced a qualifying alert within the agreed window. |
| Nuisance alerts | Non-actionable alerts per camera-day under the stated operating conditions. |
| Detection delay | Time from the defined event reference to an alert, including the range and worst observed cases. |
| Monitoring availability | How much of the observation period the specified monitoring function was operating. |
| Fault indication | Whether and how quickly lost or frozen input becomes visible to the operator. |
| Delivery and recovery | Whether the named alert endpoint receives events, and how the system behaves after a fault. |
Incident-level detection and frame-level classification are different measurements. Results should state the number of events, cameras, sites, exclusions, and uncertainty.
A staged evaluation, with a real baseline.
- Assess fit. Agree the use case, supported inputs, coverage constraints, data access, and responsibilities.
- Record the baseline. Understand how the existing camera analytics or site workflow performs.
- Configure and tune. Use representative recordings and documented operating observations.
- Evaluate separately. Keep final evaluation data distinct from tuning data. Include normal activity alongside target events.
- Review and decide. Document outcomes and limitations, then agree whether to deploy, revise, or stop.
The programme must fit the site's operating and safety procedures. Any controlled hazard testing requires an appropriate site-approved procedure and qualified personnel.
A working feed is only part of the picture.
A useful evaluation includes loss of internet, camera disconnects, frozen video, process restarts, storage limits, failed notification paths, and power recovery.
Detection, event recording, local operator access, and local output delivery should be checked independently. A central dashboard being unavailable must not be mistaken for proof that local monitoring is working—or failing.
Offline is a defined scope. Remote email, SMS, support, and synchronization may need connectivity. The actual behavior must be documented for the tested deployment.
Leave a record someone else can assess.
The proposed evaluation report identifies the tested software, configuration, camera setup, operating conditions, observation duration, and event labels. It includes successes, nuisance alerts, missed events, outages, and recovery results.
Significant configuration or software changes should lead to an appropriate re-evaluation. Independent evaluation should be considered before consequential performance claims are made.
Be explicit about what is not established.
Sentinel is in development. No universal detection rate, guaranteed early-warning time, certification, or suitability to replace an existing fire-alarm system is established.
Synthetic scenarios can support development and regression testing. They do not establish real-world reliability on their own. A rendered thermal stream does not automatically provide calibrated temperature data.
The purpose of an evaluation is to determine whether a specified configuration is useful under specified conditions—not to imply it works for every hazard or installation.
Let’s define a useful evaluation