How To Implement An Evaluation Procedure



Evaluation procedure should be in a form of a Java class, more precisely: a subclass of org.tunedit.core:EvaluationProcedure in the core package. In general, it should compare the user's answers to those expected. On that basis, the score should be computed.


Every evaluation procedure must implement the following method:

public abstract Double[] run(ResourceName solution, ResourceName dataset, ResourceLoader loader) 
                throws TunedTesterException, EvaluationSetupException, AlgorithmErrorException;

A very simple implementation could look like this:

package classification;
public class ClassificationEvalSimple extends EvaluationProcedure {
        public Double[] run(ResourceName userLabelsName, ResourceName targetLabelsName, ResourceLoader loader)
                throws TunedTesterException
                /* Open the resources */
                BufferedReader userLabels = new BufferedReader(new InputStreamReader( ));
                BufferedReader targetLabels = new BufferedReader(new InputStreamReader( ));
                try {                   
                        /* Evaluate the solution */
                        int correct = 0, count = 0;
                        String targetLabel, userLabel;
                        while ((targetLabel = targetLabels.readLine()) != null) {
                                userLabel = userLabels.readLine();
                                if (userLabel.equals(targetLabel)) //Prediction is equal to target
                        return new Double[] {correct / (double) count}; // Return the result in a one-element array of Doubles
                } catch (IOException e) {
                        throw new TunedTesterException(e);

It calculates the accuracy of the solution. Both the solution (userLabels) and test dataset (targetLabels) should be text files containing predicted/actual values.


In order to use the evaluation procedure, you have to upload it to TunedIT first.

If a procedure is to be used for a challenge, you should go to the "Evaluation" in the challenge page. In the "Evaluation procedure" section, choose "Custom" and click the "Edit" button. Next, pick a file from your computer.

Depending on the type of metric, a score can denote loss (error, i.e. the less the better) or gain (the more the better). The last step of uploading the new procedure is to choose its type and click the "Save" button.

For non-challenge evaluation procedures, simply upload them to your folder in Repository.