Multi-Objective Configuration Sampling for Performance Ranking in Configurable Systems

Yongfeng Gu, Yuntianyi Chen, Xiangyang Jia, and Jifeng Xuan
School of Computer Science, Wuhan University, Wuhan, China

1. MoConfig approach

This page shows the basic information and experimental results of MoConfig. MoConfig is a multi-objective configuration sampling strategy which aims to balances the tradeoff between the measurement cost and the ranking accuracy in the performance ranking problem. MoConfig sees the cost and accuracy as the two objectives to optimize and uses a multi-objective optimization algorithm (NSGA-II) to generate non-dominated solutions, which determines which configurations should be sampled. The details of MoConfig are depicted in Fig.1.

Fig. 1. The detailed information in MoConfig appraoch.

2. Experimental Results

We design 3 research questions and evaluate our approach on 20 datasets.

Experimental Datasets

The basic information of 20 datasets

TABLE I. The basic information of 20 datasets

Dataset name	# Options	# Configurations
rs-6d-c3-obj1	6	3840
rs-6d-c3-obj2	6	3840
sol-6d-c2-obj1	6	2803
sol-6d-c2-obj2	6	2803
wc+rs-3d-c4-obj1	3	196
wc+rs-3d-c4-obj2	3	196
wc+sol-3d-c4-obj1	3	196
wc+sol-3d-c4-obj2	3	196
wc+wc-3d-c4-obj1	3	196
wc+wc-3d-c4-obj2	3	196
wc-3d-c4-obj1	3	756
wc-3d-c4-obj2	3	756
wc-5d-c5-obj1	5	1080
wc-5d-c5-obj2	5	1080
wc-6d-c1-obj1	6	2880
wc-6d-c1-obj2	6	2880
wc-c1-3d-c1-obj1	3	1343
wc-c1-3d-c1-obj2	3	1343
wc-c3-3d-c1-obj1	3	1512
wc-c3-3d-c1-obj2	3	1512

RQ1. Can MoConfig sample fewer configurations than therank-based approach in performance ranking?

The comparison of prediction results between the rank-based method and MoConfig.

TABLE II. The number of solutions, the average measurement cost and the average RD(10) of methods

Name	Rank-based			entropy-cost			variance-cost			density-cost
Name	Solutions	Cost	RD(10)	Solutions	Cost	RD(10)	Solutions	Cost	RD(10)	Solutions	Cost	RD(10)
rs-6d-c3-obj1	50	544.12	27.82	6	134.00	18.33	100	474.73	12.04	1	93.00	13.00
rs-6d-c3-obj2	50	551.52	36.38	3	141.67	8.00	100	470.42	4.88	1	100.00	93.00
sol-6d-c2-obj1	50	406.02	35.44	4	70.25	9.00	100	344.46	3.32	2	48.50	7.00
sol-6d-c2-obj2	50	410.38	42.46	3	68.33	6.67	100	341.61	1.97	1	45.00	12.00
wc+rs-3d-c4-obj1	50	52.04	3.20	8	4.50	6.38	34	25.03	3.09	1	1.00	18.00
wc+rs-3d-c4-obj2	50	50.64	1.60	8	4.50	8.00	37	25.89	3.14	1	1.00	10.00
wc+sol-3d-c4-obj1	50	50.42	3.64	8	4.50	18.38	35	23.54	10.57	1	1.00	1.00
wc+sol-3d-c4-obj2	50	51.68	1.28	9	5.00	8.22	25	20.56	2.12	1	1.00	24.00
wc+wc-3d-c4-obj1	50	52.00	1.32	8	4.50	8.75	35	24.80	1.31	1	1.00	41.00
wc+wc-3d-c4-obj2	50	52.28	1.54	8	4.50	9.38	30	20.73	3.70	1	1.00	2.00
wc-3d-c4-obj1	50	130.52	5.12	17	10.35	29.94	91	92.80	1.59	1	1.00	16.00
wc-3d-c4-obj2	50	130.06	4.80	16	14.88	12.50	89	83.19	2.13	1	1.00	25.00
wc-5d-c5-obj1	50	175.92	23.04	7	6.00	46.71	100	126.61	3.64	1	1.00	119.00
wc-5d-c5-obj2	50	178.82	13.94	7	6.43	83.86	100	117.64	3.37	1	1.00	2.00
wc-6d-c1-obj1	50	421.78	52.70	3	66.67	29.00	100	346.36	11.28	1	45.00	25.00
wc-6d-c1-obj2	50	419.00	33.62	1	68.00	96.00	100	343.76	3.81	1	42.00	39.00
wc-c1-3d-c1-obj1	50	214.38	14.92	13	29.15	20.00	68	131.74	8.12	1	2.00	194.00
wc-c1-3d-c1-obj2	50	214.68	15.50	13	36.38	38.00	79	152.08	17.80	1	2.00	33.00
wc-c3-3d-c1-obj1	50	239.96	22.00	11	42.91	16.91	75	161.73	6.75	2	4.50	29.50
wc-c3-3d-c1-obj2	50	233.46	21.46	11	33.91	17.73	82	159.70	8.87	1	2.00	8.00

Non-dominated solutions and prediction results generated by MoConfig in each dataset.

1) rs-6d-c3-obj1

2) rs-6d-c3-obj2

3) sol-6d-c2-obj1

4) sol-6d-c2-obj2

5) wc+rs-3d-c4-obj1

6) wc+rs-3d-c4-obj2

7) wc+sol-3d-c4-obj1

8) wc+sol-3d-c4-obj2

9) wc+wc-3d-c4-obj1

10) wc+wc-3d-c4-obj2

11) wc-3d-c4-obj1

12) wc-3d-c4-obj2

13) wc-5d-c5-obj1

14) wc-5d-c5-obj2

15) wc-6d-c1-obj1

16) wc-6d-c1-obj2

17) wc-c1-3d-c1-obj1

18) wc-c1-3d-c1-obj2

19) wc-c3-3d-c1-obj1

20) wc-c3-3d-c1-obj2

RQ2. Which multi-objective optimization algorithm can beused in MoConfig?

Prediction results using different multi-objective optimization algorithms in each dataset.