Research Dataset

This page is where the Foundation publishes the raw data behind its published measurements: the numbers that the Benchmarks page reports, in the form they were collected in, so that anyone can inspect them, re-analyse them, or argue with them. A benchmark whose data nobody can inspect is an assertion, and assertions are not research.

No dataset has been published yet. When benchmark data is released, it will appear here under an open licence, with the collection method described in full. Until then, this page holds the space and explains the standard that future releases will meet.

Measurements are only as trustworthy as the method behind them, and the method is only as trustworthy as the data it produces. Publishing the raw measurements alongside the reported results closes the loop: readers can check whether the analysis was fair, whether the outliers were handled honestly, and whether the conclusions follow from the numbers.

This is reproducibility in its practical form. It does not require anyone to rebuild a laboratory; it requires that the Foundation's own data be available so that its own claims can be checked. A result that cannot be checked is a claim the reader is asked to accept on trust, and the Foundation's research programme is built on the opposite principle: everything published should be checkable.

A dataset release contains the raw measurements — the timing traces, the footprint logs, the power readings — as collected, before any summarising or smoothing. Alongside the data comes the collection method: how the measurement was set up, what was measured and what was not, the hardware configuration, the software configuration including versions, and the environmental conditions where they matter.

Where analysis scripts were used to go from raw data to reported results, those scripts are included too. The point is that a reader should be able to start from the raw data and arrive at the same results — or, if they find a different answer, to show exactly where the divergence came from.

Datasets are released under an open licence, so that anyone can download them, use them, and build on them without asking permission. The Foundation's research programme investigates problems the commercial market under-serves and publishes its findings openly rather than licensing them; the data behind the findings follows the same principle.

The specific licence for each release will be stated on the dataset page itself, alongside the data. The commitment is that it will be a genuinely open licence — no registration walls, no restrictive terms, no data that is technically public but practically unusable.

The standard for a dataset release is that the method is described fully enough for an independent reader to reproduce the measurement or to dispute the result. That includes describing what went wrong: failed runs, discarded measurements, conditions that turned out to matter. A dataset that only contains the clean runs is a curated story, not an honest record.

This is a high standard, and it means dataset releases take time to prepare. The method write-up, the configuration documentation and the data cleaning all have to be done before publication. The empty state of this page reflects that: the first release will appear when it meets the standard, not before.

The Benchmarks page is where the Foundation publishes its measurement results — the numbers, the comparisons, the conclusions. This page is the evidence behind those results. Each benchmark publication will link to the dataset release that supports it, and each dataset will link back to the benchmark write-up it belongs to.

Readers who are interested in the results should start at the Benchmarks page. Readers who want to check the results — or to use the data for their own analysis — should come here.

The first dataset release will be announced on the News page, which is the best place to watch for it. If you want to be notified directly, or if you have questions about the data programme, the contact page has the details.

And if you have collected your own measurements on embedded systems — latency, footprint, power, determinism — the Foundation is interested. Independent data makes the whole field stronger, and collaboration questions can go to partners@embeddedos.org.