IMTfolio
Documenting hands-on learning in advanced manufacturing pre-apprenticeships
Summary
This case story explores the process I undertook to solve a usability issue with IMTfolio, a multimedia portfolio that helps pre-apprentices demonstrate readiness for advanced manufacturing careers.
Through iterative cycles of testing and design, I isolated the barrier users were facing in the tool: due to insufficient built-in models and support, trainers were struggling to create goal-oriented challenges for pre-apprentices.
Successfully redesigning the user experience for trainers had a cascading effect. Trainers felt more confident using the tool and wrote better challenges. As a result, the quality of work submitted by pre-apprentices improved in kind.
Project details
Project title: Expanding Career Pathways for People with Disabilities in the Industrial Manufacturing Technician Registered Apprenticeship Program
Timeline: 2019-2020
Funder: Office of Disability Employment Policy, U.S. Department of Labor
Budget: $40,000
My role: Design lead
Background
Many jobs in advanced manufacturing require a written test as part of the hiring process. Those tests can screen out capable employees who struggle with traditional test-taking methods.
IMTfolio is a digital portfolio designed to support pre-apprentices pursuing industrial manufacturing technician (IMT) careers - giving them another way to show that they're qualified for the job.
Trainers create challenges that guide pre-apprentices to document real-world evidence of their competencies using photos and videos. The result is a multimedia portfolio showcasing an applicant performing the specific skills needed for employment.
The team discusses the STEMfolio application and its adaptations to other learning environments including IMTfolio, for advanced manufacturing pathways
Problem
The vast majority of pre-apprentice trainers in advanced manufacturing are industry professionals, sharing their first-hand knowledge and experience with the next generation of workers. It's less common for a trainer to have a background as an educator.
The success of IMTfolio hinges on trainers creating targeted, goal-oriented prompts that guide pre-apprentices to demonstrate and capture critical skills. Early on in development, we discovered the existing workflow was creating a barrier for trainers. To improve outcomes for pre-apprentices using IMTfolio, we first needed to better support trainers to develop high-quality challenges in the application.
Definition
How might we...
Support industrial manufacturing pre-apprenticeship trainers using IMTfolio to author achievable and measurable challenges tied to specific competencies
in order to...
Improve the quality of responses produced by pre-apprentices
Solution
By reworking the order of the challenge-authoring form and providing a series of show/hide models and just-in-time supports, trainers were able to more effectively create high-quality challenges in the app.
Goals
Maintain the brevity and simplicity trainers appreciated about the format
Relocate the competency selection function to increase its salience in the workflow
Build in multiple examples for reference
Rewrite the mircocopy to address the confusions trainers identified
The revised form prompts trainers to target specific competencies first and provides further clarification on their purpose in IMTfolio
Trainers can access additional support around writing effective learning goals
The use of video is encouraged to provide pre-apprentices with multiple ways to engage with information
Try out the clickable prototype ⮕ (Marvel app)
Discovery
I traveled to Milwaukee, Wisconsin to meet with participants in our pilot. After presenting an onboarding session, I conducted one-on-one interviews with four trainers. I learned that speed and simplicity were the most important factors that would impact their adoption of the tool.
With those findings in mind, we designed the trainer side to be a short, straightforward form - minimizing the steps involved to create a challenge.
In early testing, trainers reported appreciating the brevity of the form so we proceeded with the design.
Ideation
After an initial period of use, we analyzed the challenges and responses created by users. We became concerned that pre-apprentices were not concretely tying the skills they demonstrated to a particular competency.
We hypothesized that a major reason for this was the way the challenges were constructed - many were overly broad and generalized. They did not prompt pre-apprentices to explain their thinking specifically enough.
Wireframe
Version 1: Initial challenge-authoring form design
Iteration
Phase 1
In the original design, trainers were expected to write the prompt for each challenge first - without an effective way to self-assess its quality.
My first revision to the challenge-authoring form was reordering its steps. Instead of selecting competencies at the end of the process, trainers would choose them first, then write their prompt by directly referring to the competencies they chose. This simple change gave trainers a structure that produced more reliable results.
User flow
Original process
Revised process
Wireframe
Version 2: Challenge-authoring form design with revised workflow
The revised authoring process guided trainers to select competencies first, then tie the challenge goal directly to them
Phase 2
We also surmised that offering more assistance to trainers seeking clarification and examples while writing challenges would benefit both trainers and pre-apprentices. To provide this, we included just-in-time support at each stage of the authoring process. By keeping this content short and scannable, we hoped to strike a balance between brevity and utility.
Wireframe
Version 3: Challenge-authoring form design with integrated supports
The addition of just-in-time hints to the authoring form gave trainers the option to access additional support if needed
Outcomes
To test the efficacy of our revisions, we asked the trainers to re-create the challenges they had previously authored using the improved form. This allowed us to directly compare the products of each version while minimizing other variables.
We saw a marked difference in the quality and specificity of the challenges trainers created with the revised form. Challenges were more likely to refer to a particular competency and include specific, measurable goals.
A/B Test Results
Example A: 'Know your fire extinguishers' challenge using the original form
The original challenge prompt was broad and generalized; it did not guide a pre-apprentice to demonstrate evidence of a specific skill
Example B: 'Know your fire extinguishers' challenge using the revised form
The challenge prompt created with the revised form provided pre-apprentices with an achievable, goal-oriented task to perform that will demonstrate the aligning competencies