AI-based Microscopy automation
Our group leverages AI and machine learning to advance the frontiers of electron microscopy. We are actively engaged facilitating experiments with innovative beam-shaping technologies, such as the orbital angular momentum (OAM) Sorter, to achieve precise control over the electron beam; enabling the real-time optimization of experimental workflows to enhance efficiency and accuracy; and developing advanced methodologies for the analysis of large datasets, particularly in statistical and in situ experiments. Through these efforts, we integrate AI seamlessly into state-of-the-art microscopy platforms, addressing complex challenges and driving data-driven discoveries.
A long story
Our primary aim is the development of cutting-edge AI tools designed for the real-time analysis of data generated by TEM experiments. The vast amount of data produced by modern TEM techniques presents a significant challenge, creating bottlenecks in data processing and analysis. As the complexity and volume of datasets continue to grow, there is an urgent need for innovative solutions to efficiently handle this information and extract meaningful insights.
In this regard, automation through AI tools plays a crucial role in transforming how TEM experiments are conducted and analysed. By automating data interpretation, these tools offer several key advantages. First, they enable standardization across experiments, ensuring consistency and reducing human error in complex data analyses.
Moreover, the integration of AI-driven automation significantly increases the throughput of TEM experiments. Traditional methods, which often require manual data processing and interpretation by highly specialized researchers, limit the pace at which conclusions can be drawn. Automation accelerates these workflows, allowing more experiments to be processed in a shorter time, thereby unlocking the full potential of high-volume data collection techniques.
Importantly, AI-based automation also democratizes access to advanced TEM technologies. Researchers no longer need to be experts in the intricacies of electron microscopy to carry out high-level analyses. By lowering the technical barriers to entry, these tools make the technique more accessible to a broader scientific community, empowering researchers from various fields to benefit from TEM without requiring extensive specialization.
Automation
Electron beam shaping, achieved through cutting-edge technologies such as microelectromechanical systems (MEMS)-based devices, is another key focus area. These devices enable the generation of specialized beams—including vortex beams, non-diffracting beams, compact aberration correctors, and quantum state analyzers—within the microscope column.
One prominent application is the orbital angular momentum (OAM) Sorter, which decouples azimuthal and radial degrees of freedom to measure an electron beam’s OAM component. While these advancements unlock new experimental possibilities, they also introduce complexity in instrument control. To address this, we developed a convolutional neural network (CNN) capable of determining critical parameters, such as defocus and sorter electrode excitation, from a single spectrum image.
Once integrated with the microscope, this system facilitates precise tuning of the sorter within seconds, overcoming the limitations posed by manual adjustments and user skill.
Real-Time Data Analysis
The sheer volume of data produced during TEM experiments often creates bottlenecks in analysis. To overcome this, we employ advanced artificial neural networks (ANNs) to process large datasets in real time. These ANNs enable researchers to extract valuable insights almost instantaneously, allowing for faster decision-making during experiments.
To address this challenge, we have leveraged the wealth of well-established computer vision algorithms, adapting them for real-time analysis.
One notable innovation is the integration of real-time segmentation algorithms adapted from established computer vision techniques. These algorithms overlay dynamically generated insights onto the live video feed of the microscope, creating an augmented reality environment. This approach enhances the user’s perception of the sample, highlights relevant regions in real time, and streamlines the discovery process, making workflows more responsive and data-driven.
Optimisation
AI plays a crucial role in optimizing the trajectory of TEM experiments. Predictive models based on ANNs evaluate early, partial measurements to forecast the most likely outcomes of ongoing experiments. This capability allows researchers to adjust experimental parameters in real time, maximizing resource efficiency and reducing the number of iterations needed to achieve desired results.
A compelling example is Computation Ghost Imaging (CGI). Unlike conventional TEM imaging methods, where an image is formed by either capturing beam intensity on an array of detectors (camera) or by raster scanning a point-like electron beam across the sample, CGI offers an alternative approach relies on a series of repeated measurements, illuminating the sample with a structured illumination pattern rather than a continuous beam, and by collecting the whole transmitted intensity with a single (bucket) detector. CGI uses a relatively low-cost electron beam modulator to create these structured patterns, which provides a more cost-effective alternative to expensive electron cameras or probe correctors typically required for point-like illumination.
While a wide range of machine learning methods exist to predict the next optimal illumination patterns—including Gaussian processes, discrete kernel learning, active learning, deep Q-learning, and Monte Carlo Tree Search—a key challenge in this domain is accurately predicting the shape of the electron beam pattern generated by the electron beam modulator. This modulator, typically a small MEMS device positioned along the beam path, can introduce variability in the patterns, which needs to be accounted for in order to ensure precise and efficient selection of the next pattern. Predictive models must therefore not only optimize the information gain from each pattern but also factor in the physical limitations and variability of the modulator itself to enhance the accuracy of the overall image reconstruction process.
To predict the shape of the illumination pattern produced by this modulator when biased, we employed a NN that takes the nominal bias values of the needles as input and outputs the corresponding illumination pattern.
