Preliminary vs. Final Evaluation

 

 

For the sake of full objectivity, there are two distinct stages of evaluation in TunedIT Challenges: preliminary and final. Preliminary evaluation is run during the contest, every time when the participant submits improved solution, possibly many times for a given participant. Result of the active solution - the last or the best one (depending on challenge settings) submitted so far - is shown always on the Leaderboard. Final evaluation is executed only once, at the end of the contest, for the active solution.

All the settings can be configured differently for both stages: evaluation procedure, time limit, precision of results printed on Leaderboard, and - most importantly - the datasets. Two disjoint subsets of full data can be used for preliminary and final tests, which guarantees that final solutions - no matter if the Last or Best preliminary solution is selected as the final one - will be evaluated in objective way and final results won't be biased. Note that the final solution - which is the active solution at the end of the challenge - is identified on the basis of preliminary results of all submissions, not the final results.

See also: List of Features of TunedIT Challenges