Learning technology piloting: building momentum with our first working group
Back in March, in collaboration with University of Exeter, Dr Steph Comley and I held the first meeting of our Learning Technology Piloting consultation group. We brought together educators, digital learning specialists and senior leaders from across UK tertiary education to reflect together on piloting learning technology.
Our goal for the session was to start sharing experiences of piloting learning technology:
- What goes well and why?
- When things don’t work out, what do we learn?
- What are the gaps and what often gets overlooked?
In practice, the conversation quickly opened into much wider territory. We talked about language, governance, workload, evaluation and what pilots actually feel like for staff and students.
What followed was an honest, thoughtful and lively discussion. Below, I’ve pulled together the main themes, structured around what tends to matter before, during and after a pilot, and what we think should come next.
Before a pilot
One of the strongest messages from the group was the value of spending time up front, even when there’s pressure to ‘just try something.’
Start with the problem (not the shiny tool)
Many of us recognised the pull of a new or promising tool. But pilots are much more likely to be useful when they start with a clearly defined problem or opportunity, rather than curiosity alone. Bringing in the student perspective early was seen as particularly important here.
How do we make sure it actually fits?
Strategic alignment came up again and again. If a pilot does not clearly support institutional priorities, it is much harder to justify the time, effort and risk involved and then complicates decision making at the end.
Be clear about what kind of thing this is
We spent time untangling terms like pilot, prototype and proof-of-concept. These words carry expectations, whether we mean them to or not. Being explicit about scope, scale and success criteria helps avoid confusion, especially when interest in a tool starts to grow.
Bring governance in early
Data protection, accessibility, equality impacts, procurement, ethics and sustainability: these areas need to be considered and relevant teams need to be brought in early. Participants stressed how much easier pilots are when professional services colleagues are involved early, shaping the work rather than being asked to validate decisions that have already been made.
Be honest about cost and capacity
Several people spoke about pilots being described as ‘lightweight’ when, in reality, they are anything but. Staff time, project management, training, support and potential long-term costs all need to be factored in. For a pilot to be successful, a much larger number of stakeholders may need to be involved than initially expected. Capacity pressures on digital and learning technology teams are real, and pilots often land there by default. It was also noted that it is essential to review existing tool functionality first before proposing new tools particularly given current sector financial challenges.
Plan evaluation properly
Evaluation works best when it is designed early and treated seriously. The group shared a lot of concern about survey overload and performative reporting. Clear questions and proportionate evidence-gathering make a big difference.
Consult existing frameworks
People shared existing frameworks to help in areas such as digital transformation, user acceptance, change management, project lifecycle management, EdTech evaluation and understanding organisational effectiveness. A RACI (responsible, accountable, consulted, informed) matrix or similar could be used for the crucial step of defining roles and responsibilities. Decisions have to be made to decide which frameworks can support you and plan them into your project roadmap.
During a pilot
When a pilot begins, more challenges start to appear.
Project management
Dedicated project management came up as a huge benefit. However, having a dedicated project manager is not a given particularly in the current climate. Where it does not, coordination work often quietly lands with digital learning teams, on top of everything else. This also makes it harder to coordinate across multiple teams.
Engagement does not look after itself
Pilots often lose momentum not because anyone has done anything wrong, but because people are busy. Regular communication, realistic expectations and visible care for participants help keep things on track and maintain trust. This relates to the project manager point above.
Look for learning as you go
Rather than waiting until the end, participants shared positive experiences of gathering feedback throughout the pilot. Short reflections or structured tasks can surface issues early and produce richer insights than a single end-of-pilot survey.
Pay attention to vendor behaviour
Experiences with vendors varied widely. Responsiveness, support quality and long-term sustainability all matter. Several people noted that intensive pilot support often disappears later unless paid for. Being confident you can extract your data at any point was seen as non-negotiable.
Beware the scope creep
Pilots need room to adapt when things change. At the same time, expanding scope mid-way can undermine evaluation and decision-making. Managing this tension is part of the work.
After a pilot
The “after” stage generated some of the strongest feelings in the room.
Tell people what happened
Clear communication is key. Telling people what was decided and why. Silence can leave people feeling that their time and effort were wasted, even if the pilot itself was positive.
It’s OK to say no
A pilot going well does not automatically mean a tool should be adopted. The group shared examples where saying no was the right decision, once wider risks, costs and impacts were considered. Being able to stop is a sign of good governance, not failure.
Handle access and data carefully
There needs to be clarity about what happens to accounts, data and access once a pilot ends. Planning for closure from the start helps avoid rushed or uncomfortable endings later on.
Acknowledge people’s contribution
Taking part in pilots requires trust and goodwill. Recognising that time and effort helps build a culture where people are willing to engage again in the future.
What next?
Across the discussion, there was a strong appetite for shared, practical guidance. People also asked for space to share cautionary tales and lessons learned, not just success stories.
This consultation group is an early step in that direction. Our next focus is on turning these insights into a resource to help people and teams throughout the whole piloting process.
Get involved
If you are involved in piloting learning technology, or are affected by it, we’d really like to hear from you. That might be as an educator, a digital learning specialist, a leader, a student or someone who has taken part in multiple pilots with mixed experiences.
You could:
- Share examples of what has helped or hindered pilots in your context
- Challenge our assumptions and fill in the gaps
- Contribute case studies or questions to help shape future guidance
Interested in shaping what comes next? Get involved in future consultation sessions or share your insights by contacting Cat Bailey cat.bailey@jisc.ac.uk. Your input will directly inform practical guidance grounded in the real-world experience of piloting learning technology across the sector.
Pilots are often framed as low risk or importance, but for the people involved they rarely feel that way. Learning from each other is one of the best ways we have to make them more thoughtful, fair and genuinely useful.
Thank you to all who were involved in the first consultation group and those who have already kindly shared their knowledge, resources and experiences.
Header image: A tech project is like…by Visual Thinkery is licensed under CC-BY-ND