liu.seSearch for publications in DiVA
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • oxford
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
IRIS: Iterative and Intelligent Experiment Selection
University of Calgary Calgary, Canada.
Linköping University, Department of Computer and Information Science, Database and information techniques. Linköping University, Faculty of Science & Engineering.
University of Calgary, Calgary, Canada.
University of Calgary Calgary, Canada.
2017 (English)In: ICPE ’17 Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering, ACM , 2017, p. 143-154Conference paper, Published paper (Refereed)
Abstract [en]

Benchmarking is a widely-used technique to quantify the performance of software systems. However, the design and implementation of a benchmarking study can face several challenges. In particular, the time required to perform a benchmarking study can quickly spiral out of control, owing to the number of distinct variables to systematically examine. In this paper, we propose IRIS, an IteRative and Intelligent Experiment Selection methodology, to maximize the information gain while minimizing the duration of the benchmarking process. IRIS selects the region to place the next experiment point based on the variability of both dependent, i.e., response, and independent variables in that region. It aims to identify a performance function that minimizes the response variable prediction error for a constant and limited experimentation budget. We evaluate IRIS for a wide selection of experimental, simulated and synthetic systems with one, two and three independent variables. Considering a limited experimentation budget, the results show IRIS is able to reduce the performance function prediction error up to 4:3 times compared to equal distance experiment point selection. Moreover, we show that the error reduction can further improve through system-specific parameter tuning. Analysis of the error distributions obtained with IRIS reveals that the technique is particularly effective in regions where the response variable is sensitive to changes in the independent variables

Place, publisher, year, edition, pages
ACM , 2017. p. 143-154
Keywords [en]
controlled experimentation, performance benchmarking, system performance
National Category
Computer and Information Sciences
Identifiers
URN: urn:nbn:se:liu:diva-140915DOI: 10.1145/3030207.3030225ISBN: 978-1-4503-4404-3 (print)OAI: oai:DiVA.org:liu-140915DiVA, id: diva2:1141629
Conference
8th ACM/SPEC on International Conference on Performance Engineering, L'Aquila, Italy, April 22-26, 2017
Available from: 2017-09-15 Created: 2017-09-15 Last updated: 2018-01-13

Open Access in DiVA

fulltext(811 kB)42 downloads
File information
File name FULLTEXT01.pdfFile size 811 kBChecksum SHA-512
6c7ea6d46e24812cfe720def68cdb87438508c333e680d65c12a660b6155cbc06db180a321f713aa44e1bc851b3e1024cd1756a028709f9e46968b258da3a98e
Type fulltextMimetype application/pdf

Other links

Publisher's full text

Search in DiVA

By author/editor
Carlsson, Niklas
By organisation
Database and information techniquesFaculty of Science & Engineering
Computer and Information Sciences

Search outside of DiVA

GoogleGoogle Scholar
Total: 42 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

doi
isbn
urn-nbn

Altmetric score

doi
isbn
urn-nbn
Total: 53 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • oxford
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf