The Dedoose Training Center is all about great research and our efforts to include features that support the development and practice of systematic and rigorous qualitative and mixed methods research. After a lot of hard work by our Dedoose team, we’re thrilled to finally see the Training Center online. The Training Center is, as one user put it, ‘a god-send’ for both research individuals and teams who want to insure reliability and validity in their qualitative code system application and quantitative code weighting activity. We began by drawing upon our pioneering implementations in EthnoNotes, and then expanded the Training Center to increase the efficiency of creating tests, reviewing test results, and implemented more rigorous and sophisticated statistics and other test feedback. We are confident that users will find the system a pleasure to use and will appreciate how the Center helps researchers build and maintain scientifically valid and rigorous best practices.
After entering the training workspace (Fig. 1), click ‘Create New Test.’ A popup will appear and prompt you to select which type of training test you would like to create, (Fig. 2), a test to develop and assess consistency in code application or a test to develop and assess consistency in setting weights/ratings for codes that have been applied. Let’s start by discussing the code application test. Click ‘Code Application Test’ and then ‘Next.’ Now you will want to select the codes you want to use for the purpose of your test (Fig. 3) and click ‘Next.’ Note that at this point you can click back at any time and change all your inputs until you finalize your test. On the Excerpt Selection screen you will want to select the excerpts that will comprise your test—these are the excerpts that the trainee will be review and code. The excerpt selector automatically limits the available excerpts to those that contain at least one of the codes you selected to include in the test. The excerpt selector is the same as many other work spaces in Dedoose, so you have all the options to customize the view, sort, and filter. Once you have the excerpts selected that you would like to include, click ‘Next.’ Finally you are presented with a screen to title and describe the test (Fig. 4)—and you’ll want to provide useful information so it will be easy to identify in the library of tests you create. Once you’ve filled out these details, ‘Save’ to complete your test and return to the Training Center home screen.
Once you have a test in your library, clicking on it brings up the test detail window (Fig. 5). Here we can see the overall results for anyone who has taken this test, we can permanently delete the test and its associated results, or we can take the test….let’s do that. Click ‘Take this test’ to begin. Since this is a code application test, we will now see the excerpts we have selected (without any of the codes that have been applied), and be presented with the codes that were enabled for this test (Fig. 6). The trainee/test taker can now double click any code to apply that code to the current excerpt, and use the next and back buttons to move between all the excerpts in this test. On the last excerpt you will notice the finish button, which, when clicked prompts you to complete this test and save the result. Upon finishing the test you will be brought immediately to the test review window (Fig. 7). In this window you will be presented with the overall test result, ‘Pooled Cohen’s kappa.’ This is a respected calculation of inter-rater agreement across multiple codes. This value can range from 0 to 1, with values closer to 1 indicate stronger agreement. On this window we provide additional information on the history of Cohen’s Kappa and its implementation in the Dedoose Training Center (including the evaluation of meaningful/useful results and associated research references). In the lower portion of the window you will find the detailed information on each code in the test, including the number of agreements and disagreements for each code across all excerpts, as well as the individual Cohen’s kappa statistic for each code with an accompanying visual color indicator to show the approximate ‘success’ of the trainee for the particular code.
Further, to examine the specific results for each excerpt, click ‘View Code Application’ to view each excerpt and view, side-by-side, the codes that were applied by both the ‘trainer’ and the ‘trainee’ (Fig. 8). This detailed information is ideal for the conversations among team members to evaluate the validity of codes and their application criteria with the ultimate goal of a code system that can be meaningfully and consistently applied by all team members. Once this system and systematic application criteria are identified and documented, the team is ready to confidently move forward with rigorous qualitative data coding.
The process for creating and taking a code weight/rating test is very similar to that described for code application. When selecting codes for the test, only those codes with associated weight/rating systems will be available for inclusion in a test. And when taking the test, the trainee is presented with the excerpts, the applied codes and the default value for the weight/rating scale. The trainee’s task is then to modify weight/rating value to what they believe is appropriate for the particular excerpt. The Code Weighting Test result window (Fig. 9) looks similar to the code application test result window, but the measure of association being Pearson’s correlation coefficient to represent the degree of agreement between the weighting/rating applications. And, again, you can walk through the test result excerpts and view a side-by-side comparison of the ‘trainer’ and ‘trainee’s’ weightings/ratings for further evaluation. Finally, as a parametric statistic, we provide a link to a table of critical values for reporting the statistical significance of the observed Pearson’s correlation coefficient.
The Dedoose Team
;