Click here for a document about SkillTRAN Reliability, Validity, and Peer Review
The information below is extracted from the full document available in the above link.
SSA Validation Study of TSA Process
Using the transferable skills technology it developed for the U.S. Social Security Administration (SSA), SkillTRAN’s predecessor (George M. Watters of Ability Information Systems) built a highly specialized computer program in the mid-1980s called VARS, the Vocational Adjudication Rehabilitation System. In a multi-year, multi-state study of Computer Assisted Vocational Evaluation (CAVE) systems (SSA, 1987), VARS was shown to emulate the manual process followed by disability examiners at Steps 4 and 5 of its 5-Step process in the SSA sequential evaluation process. The thousands of cases processed by VARS and by a manual control group showed identical rates of allowance and denial at Steps 4 and 5. Such a result supports essentially identical concurrent validity of the VARS process with the SSA manual process. The study was designed to examine faster case processing rather than validity, so no special calculations were done beyond reporting of percentages of claims decisions, test vs. control. Quality review by an independent internal SSA review process showed substantially fewer disagreements with the cases processed by VARS than with vocational evaluation cases that had been manually processed by regular SSA staff.
Peer Reviewed Since 1983
In addition to the implicit concurrent validity study by SSA in 1987, various products now owned by SkillTRAN LLC have been independently reviewed by Karl Botterbusch (1983, 1986). These software systems were again reviewed by an independent group in the book Vocational Evaluation Systems and Software: A Consumer’s Guide (Brown, McDaniel, Couch, & McClanahan, 1994). Consumer ratings for the SkillTRAN software products (then known as CAPCO) showed the highest consumer ratings for Comprehensiveness and Overall Satisfaction (p. 137).
Software Used by SSA
The SSA uses licensed copies of the Job Browser Pro, OccuBrowse, and OASYS software available from SkillTRAN in its electronic digital library. The software is recognized by SSA as an acceptable electronic version of the DOT and as “useful electronic reference tools.” (SSA, 2009) These three software applications consistently rank as among the top 5 most used commercial applications within SSA.
Job Numbers
SkillTRAN uses a unique method for refining the estimation of government employment data down to the level of the DOT occupation itself. Since its introduction in 2008, this process has become widely used by both Vocational Experts and Claimant Representatives in appeal hearings for SSA disability benefits eligibility. The SkillTRAN Process for Estimating DOT Employment Numbers methodology is disclosed in this linked document, which has been continuously published at www.skilltran.com since 2008. This estimation process is also finding favor in vocational rehabilitation planning efforts for injured workers in the workers’ compensation industry. Click here for a detailed article on Job Numbers.
The late Joe Havranek, initially skeptical in his 2008 published review of the software, saw that the process closely emulated his expertly derived method (Havranek, 2008).
Amberly Ruck (2014) wrote a brief book explaining her experiences using this new method in the SSA venue, first as a vocational expert and now as a non-attorney claimant representative.
SkillTRAN’s approach to estimating employment numbers at the DOT level has also been carefully reviewed by several attorneys, including Laurence Rohlfing (2013) and Peter Lemoine (2012). Former Administrative Law Judge/Current Attorney Claims Representative Lemoine’s review goes so far as to suggest that the estimation process developed by SkillTRAN may well withstand a Daubert evaluation. Such a suggestion supports the notion that the estimation process falls within reasonable probability and constitutes substantial evidence. While SSA is bound neither by Daubert criteria nor by Federal Rules of Evidence (Brault vs. SSA, 2012), SkillTRAN continues to move towards procedures that would withstand a Daubert challenge.
Another important issue is the concept of “peer review”. Traditional peer review includes critical written analysis presented in formal professional journals. Lemoine’s comments about SkillTRAN/Job Browser Pro are a good example, as his article was published in three parts in the newsletter of the National Organization of Social Security Claimant Representatives (NOSSCR). [NOTE: NOSSCR endorses no products.]
Alternate Methods of Review by Peers
SkillTRAN recognizes the importance of peer review and adds another essential dimension to the concept: the day-to-day use of its products by customers in many venues, including litigated and non-litigated environments such as rehabilitation, SSA, Workers Compensation, and civil proceedings. Moreover, SkillTRAN encourages vocational and forensic experts to modify the industries suggested by SkillTRAN based on their professional experience. Rather than deliver a static process, SkillTRAN embraces the dynamic expertise of its customers and enables them to easily contribute their practical and current knowledge of industry trends.
While some may question the ability of the user to make changes since these changes impact the total estimate of job numbers, SkillTRAN believes this feature is one of its core strengths. SkillTRAN will analyze the user-suggested DOT to NAICS industry changes and modify its “master” cross-reference file based on appropriate customer feedback. This revised master cross-reference file will then be periodically released back to customers to reflect not only the industries suggested by SkillTRAN, but those suggested by its overall customer base throughout the US and its territories. In today’s culture of social media and rapidly changing data, what better, stronger and more dynamic review can there be than continuous customer review?
REFERENCES
Brault vs. Social Security Administration Commissioner, 683 F.3d 443 (2nd Cir. 2012).
Botterbusch, K. F. (1983). A comparison of computerized job matching systems. Menomonie, WI: Materials Development Center.
Botterbusch, K. F. (1986). A comparison of computerized job matching systems (2nded.). Menomonie, WI: Materials Development Center.
Brown, C., McDaniel, R., Couch, R., & McClanahan, M. (1994). Vocational evaluation systems and software: A consumer’s guide. Menomonie, WI: Materials Development Center.
Lemoine, P. J. (2012). Crisis of confidence: The inadequacies of vocational evidence presented at Social Security disability hearings. Printed in three parts in the newsletter of the National Organization of Social Security Claimant Representatives. Retrieved from: https://www.lb7.uscourts.gov/documents/15cv1230URL11LemoineLawseminarmaterials.pdf
Rohlfing, L. (2013, November). Confessions of a lawyer in transition. How I changed my thinking about numbers of jobs. Handout from The numbers game: How vocational experts derive employment numbers of occupations. Presented at the Fall Conference of the National Association of Social Security Claimant Representatives, San Diego, CA.
Rohlfing, L. & Truthan, J. A. (2013, November). The numbers game: How vocational experts derive employment numbers of occupations. Presented at the Fall Conference of the National Association of Social Security Claimant Representatives, San Diego, CA.
Ruck, A. M. (2014). Why have the judges fallen? Expert witness testimony of corruption in Social Security disability adjudications. San Bernardino, CA: Author.
Social Security Administration, Associate Commissioner for Disability (1987). Letter from David Rust to Ability Information Systems dated November 6, 1987.
Social Security Administration, Division of Field Services. (2009). Memorandum 09-2139 dated 12/28/2009 from Susan Swansiger, Director. Subject: Use of Electronic Occupational References for Administrative Law Judges and Senior Attorney Adjudicator Decisions – UPDATE.
U.S. Department of Labor. Manpower Administration. (1972). The handbook for analyzing jobs. Washington, DC: U. S. Government Printing Office.
U.S. Department of Labor. (1977). Dictionary of occupational titles (4th ed.). Washington, DC: U.S. Government Printing Office.
U.S. Department of Labor. (1991a). Dictionary of occupational titles (4th ed. revised). Washington, DC: U.S. Government Printing Office.
U.S. Department of Labor. (1991b). The revised handbook for analyzing jobs. Washington, DC: U.S. Government Printing Office.