TriScale

TriScale_logo

A Framework Supporting Replicable Performance Evaluations in Networking

Following a live tutorial session? Here are the links you’re looking for

Hands-on
Part 1
Part 2
Notebook Basics

Tell us about your experience!
If you followed one of our tutorials on TriScale, we would love to hear what you thought about it!
🙏 Please fill out this short survey (5min max)

👉

When designing their performance evaluations, networking researchers often encounter questions such as:

How long should a run be?
How many runs to perform?
How to account for the variability across multiple runs?
What statistical methods should be used to analyze the data?

Despite the best intentions, researchers often answer these questions differently, thus impairing the replicability of evaluations and the confidence in the results.

Improving the standards of replicability has recently gained traction overall, as well as within the networking community. As an important piece of the puzzle, we developed a systematic methodology that streamlines the design and analysis of performance evaluations, and we have implemented this methodology into a framework called TriScale.

Reproduce our study

All data, code and information required to reproduce our study are openly available on GitHub.

We use Jupyter notebooks as wrappers for all our analyses. You can reproduce the paper results by running the notebooks locally or directly in your web-browser, thanks to the amazing service provided by MyBinder.org!

See the README.md file on GitHub for detailed instructions.

Presentations

For more resources, check out the TriScale’s tutorial page.

Research replicability in embedded learning
ESWEEK Education, Virtual (October 2021)

Tutorial: Supporting Replicable Networking Experiments with TriScale
ACM SIGCOMM 2021, Virtual (August 2021)

Tutorial: Supporting Replicable Networking Experiments with TriScale
4th CPS-IoTBench Workshop, Virtual (May 2021)

IoTBench: Reproducibility challenge in wireless networking research
5th CROSS Symposium, Virtual (October 2020)

Confidence in experimental evaluations:
Time to do better than “Believe me, it’s true!”
EWSN Conference, Lyon, France (February 2020)

How to cite TriScale

@proceedings{jacob2021triscale,
  title         = {Designing Replicable Networking Experiments with TriScale},
  author        = {Jacob, Romain and Zimmerling, Marco and Boano, Carlo Alberto and Vanbever, Laurent and Thiele, Lothar},
  year          = 2021,
  doi           = {10.5070/SR31155408},
  year          = {2021},
  month         = nov,
  journal       = {JSys},
  volume        = {1},
  number        = {1},
  issn          = {2770-5501},
  url           = {https://escholarship.org/uc/item/63n4s9w2},
  urldate       = {2021-11-15},
  area          = {Networking},
  artifacts_url = {https://github.com/romain-jacob/triscale},
  langid        = {english},
  review_url    = {https://openreview.net/forum?id=c1LNi8CTPy6},
}

People
Romain Jacob
Marco Zimmerling
Carlo Alberto Boano
Laurent Vanbever
Lothar Thiele