材料科學(xué)主要研究材料的結(jié)構(gòu)與性質(zhì)之間的關(guān)系，這些關(guān)系跨越從原子到微米尺度。掃描透射電子顯微鏡（STEM）已成為在這些尺度上研究材料的重要工具，特別是由于其能夠與先進(jìn)的數(shù)據(jù)分析技術(shù)相結(jié)合，為自動(dòng)化實(shí)驗(yàn)和多維數(shù)據(jù)處理提供了新的機(jī)遇。

Fig. 1 Example of the errors introduced via StyleGAN116.

隨著機(jī)器學(xué)習(xí)算法的發(fā)展，STEM在實(shí)時(shí)分析和自動(dòng)控制方面的應(yīng)用前景廣闊。由美國(guó)田納西大學(xué)材料科學(xué)與工程系的Sergei V. Kalinin教授和橡樹(shù)嶺國(guó)家實(shí)驗(yàn)室計(jì)算科學(xué)與工程部的Debangshu Mukherjee博士領(lǐng)導(dǎo)的團(tuán)隊(duì)，對(duì)掃描透射電子顯微鏡中的自動(dòng)化實(shí)驗(yàn)機(jī)器學(xué)習(xí)進(jìn)行了綜述。

Fig. 2 Automated task-based statistical analysis via few-shot learning.

掃描透射電子顯微鏡及其光譜技術(shù)已經(jīng)成為現(xiàn)代材料科學(xué)、凝聚態(tài)物理、化學(xué)和生物學(xué)等領(lǐng)域的基石工具。這項(xiàng)技術(shù)的影響力與其能夠洞察材料結(jié)構(gòu)和性質(zhì)的量化信息量直接相關(guān)。無(wú)論是冷凍電子顯微鏡（Cryo EM）還是小晶體電子衍射等領(lǐng)域的突破都表明，數(shù)據(jù)分析方法和高效的操作流程極大地提高了從技術(shù)發(fā)展中所獲得的價(jià)值，并顯現(xiàn)出該領(lǐng)域的巨大增長(zhǎng)潛力。

在STEM領(lǐng)域，自動(dòng)化實(shí)驗(yàn)的發(fā)展是一個(gè)快速崛起的趨勢(shì)。目前，正從人工實(shí)驗(yàn)向自動(dòng)化實(shí)驗(yàn)的過(guò)渡期中，面臨著眾多挑戰(zhàn)。在儀器方面，需要發(fā)展高級(jí)別超語(yǔ)言，以便用最基礎(chǔ)的操作單元描述人類動(dòng)作。在機(jī)器學(xué)習(xí)領(lǐng)域，這要求開(kāi)發(fā)出對(duì)分布外漂移效應(yīng)具有魯棒性的監(jiān)督學(xué)習(xí)算法，以及能夠在少量數(shù)據(jù)上訓(xùn)練的主動(dòng)學(xué)習(xí)技術(shù)。

在計(jì)算和網(wǎng)絡(luò)領(lǐng)域，這需要構(gòu)建邊緣計(jì)算基礎(chǔ)設(shè)施，不僅能夠支持快速分析和決策，還能將儀器接入全球云網(wǎng)絡(luò)。這一點(diǎn)將進(jìn)一步推動(dòng)高效的數(shù)據(jù)與代碼共享，形成分布式的人機(jī)協(xié)作團(tuán)隊(duì)，并催生出跨儀器的網(wǎng)絡(luò)協(xié)作平臺(tái)。

盡管如此，向自動(dòng)化實(shí)驗(yàn)的轉(zhuǎn)變同樣要求科學(xué)界在規(guī)劃與實(shí)施實(shí)驗(yàn)活動(dòng)方面作出根本性的改變。目前為止，已知的所有顯微鏡自動(dòng)化實(shí)驗(yàn)均采用基于固定策略和預(yù)先定義的興趣對(duì)象的工作流。僅有的超越傳統(tǒng)人工操作流程的實(shí)驗(yàn)案例，是那些基于深度核學(xué)習(xí)的逆向發(fā)現(xiàn)實(shí)驗(yàn)。

要真正釋放自動(dòng)化實(shí)驗(yàn)的潛能，關(guān)鍵在于明確定義實(shí)驗(yàn)激勵(lì)，即明確的實(shí)驗(yàn)?zāi)繕?biāo)，這可以是探索性發(fā)現(xiàn)、假設(shè)驗(yàn)證或定量測(cè)量等。許多這樣的激勵(lì)目標(biāo)通常只在特定領(lǐng)域應(yīng)用的更寬廣的科學(xué)背景中才能被界定。接下來(lái)，需要制定確定性或概率性的策略，即將以超語(yǔ)言表達(dá)的具體行動(dòng)與系統(tǒng)的當(dāng)前狀態(tài)（圖像或光譜）連接的算法。這些策略可以在實(shí)驗(yàn)前設(shè)定，以協(xié)調(diào)探索和利用之間的目標(biāo)，或者更引人注目的是，策略可以隨著實(shí)驗(yàn)的進(jìn)行而不斷發(fā)展，以便在既定實(shí)驗(yàn)預(yù)算內(nèi)實(shí)現(xiàn)既定的獎(jiǎng)勵(lì)目標(biāo)。

綜上所述，STEM領(lǐng)域的自動(dòng)化實(shí)驗(yàn)（AE）雖處于起步階段，但變化迅猛。鑒于基于Python的API和云基礎(chǔ)設(shè)施、遠(yuǎn)程控制的顯微鏡的迅速發(fā)展，尤其是考慮到貝葉斯優(yōu)化、強(qiáng)化學(xué)習(xí)以及其他隨機(jī)優(yōu)化形式等主動(dòng)學(xué)習(xí)方法的最新進(jìn)展，可以預(yù)見(jiàn)該領(lǐng)域?qū)⒃谖磥?lái)幾年內(nèi)迎來(lái)快速增長(zhǎng)。

該文近期發(fā)表于npj Computational Materials 9: 227 (2023).

Editorial Summary

Materials science focuses on the study of the relationships between the structure and properties of materials that span from the atomic to the micrometer scale. Scanning transmission electron microscopy (STEM) has become an important tool for studying materials at these scales, especially due to its ability to be combined with advanced data analysis techniques, which provide new opportunities for automated experiments and multidimensional data processing. With the development of machine learning algorithms, STEM has promising applications in real-time analysis and automation.?

A team lead by Prof. Sergei V. Kalinin from Department of Materials Science and Engineering, University of Tennessee and Dr. Debangshu Mukherjee from Computational Sciences and Engineering Division, Oak Ridge National Laboratory, USA, reviewed machine learning for automated experimentation in scanning transmission electron microscopy. Scanning transmission electron microscopy and spectroscopy has become one of the foundational tools in modern materials science, condensed matter physics, chemistry, and biology. The impact of this technique is directly related to the amounts of quantifiable information on materials structure and properties it can derive. The success of fields such as Cryo EM and small crystal electron crystallography suggest that the availability of the data analysis methods and operational workflows greatly amplifies the value derived from technique developments and suggests tremendous potential for the field growth. One of the rapidly emerging trends in STEM is the development of the automated experiments.?

Here, the authors overview some of the challenges that transition from human-driven to automated experiment EM will bring. On the instrument side, this necessitates the development of the instrument-level hyper-languages that allow to represent the human operations via minimal primitives. On the ML side, it requires development of the supervised ML algorithms that are stable with respect to the out of distribution drift effects and active learning methods that can be trained on small volumes of data. On the computational and network side, it requires development of edge computing infrastructure capable of supporting rapid analysis and decision making, and connect the instrument to the global cloud. The latter in tern opens the pathway to the effective data and code sharing, formation of the distributed human-ML teams, and emergence of the lateral instrumental networks. However, the transition to the automated experiments also requires deep changes in the way scientific community plans and executes experimental activities. To date, all examples of the automated experiment in microscopy the authors are aware of are performed with the workflows based on fixed policies and a priori known objects of interest. The only examples of beyond human workflows include the inverse discovery experiments based on the deep kernel learning. Going beyond simple imitation of human operation and unleashing the power of automated experiment requires clearly defining the experimental reward, i.e. specific goals. This can include the discovery (curiosity learning), hypothesis falsification, or quantitative measurements. Many of these rewards are defined only within a broader scientific context of specific domain applications. Secondly, this requires formulating the deterministic or probabilistic policies, i.e. algorithms connecting the specific action expressed in the hyper language and the observed state of the system (image or spectra). These policies can be defined prior to the experiment to balance the exploration and exploitation goals. Alternatively, and much more interestingly, the policies can evolve along the experiment to achieve the desired reward within the given experimental budget.?

Overall, the current state of the AE in STEM is nascent but fast changing. However, given the rapid emergence of the Python-based APIs and cloud infrastructure, remotely controlled microscopes, and especially given recent advances in active learning methods including Bayesian Optimization, reinforcement learning, and other forms of stochastic optimization, this field is likely to grow quickly in the coming years.

This review article was recently published in npj Computational Materials 9: 227 (2023).

原文Abstract及其翻譯

Machine learning for automated experimentation in scanning transmission electron microscopy（機(jī)器學(xué)習(xí)在掃描透射電子顯微鏡自動(dòng)實(shí)驗(yàn)中的應(yīng)用）

Sergei V. Kalinin,?Debangshu Mukherjee,?Kevin Roccapriore,?Benjamin J. Blaiszik,?Ayana Ghosh,?Maxim A. Ziatdinov,?Anees Al-Najjar,?Christina Doty,?Sarah Akers,?Nageswara S. Rao,?Joshua C. Agar?&?Steven R. Spurgeon?

Abstract?Machine learning (ML) has become critical for post-acquisition data analysis in (scanning) transmission electron microscopy, (S)TEM, imaging and spectroscopy. An emerging trend is the transition to real-time analysis and closed-loop microscope operation. The effective use of ML in electron microscopy now requires the development of strategies for microscopy-centric experiment workflow design and optimization. Here, we discuss the associated challenges with the transition to active ML, including sequential data analysis and out-of-distribution drift effects, the requirements for edge operation, local and cloud data storage, and theory in the loop operations. Specifically, we discuss the relative contributions of human scientists and ML agents in the ideation, orchestration, and execution of experimental workflows, as well as the need to develop universal hyper languages that can apply across multiple platforms. These considerations will collectively inform the operationalization of ML in next-generation experimentation.

摘要 機(jī)器學(xué)習(xí)（ML）已成為（掃描）透射電子顯微鏡(S)TEM成像和光譜數(shù)據(jù)后期處理的關(guān)鍵技術(shù)。目前的一個(gè)新趨勢(shì)是向?qū)崟r(shí)分析和閉環(huán)顯微鏡操作的過(guò)渡。在電子顯微鏡中有效利用機(jī)器學(xué)習(xí)現(xiàn)在需要開(kāi)發(fā)以顯微鏡為中心的實(shí)驗(yàn)工作流程設(shè)計(jì)和優(yōu)化策略。在這里，我們討論了向主動(dòng)機(jī)器學(xué)習(xí)過(guò)渡所面臨的挑戰(zhàn)，包括順序數(shù)據(jù)分析、分布外漂移效應(yīng)、邊緣運(yùn)算要求、本地和云數(shù)據(jù)存儲(chǔ)，以及環(huán)路理論操作。特別是，我們討論了人類科學(xué)家和機(jī)器學(xué)習(xí)代理在實(shí)驗(yàn)工作流程的構(gòu)思、協(xié)調(diào)和執(zhí)行中的相對(duì)貢獻(xiàn)，以及開(kāi)發(fā)可跨多個(gè)平臺(tái)應(yīng)用的通用超級(jí)語(yǔ)言的必要性。這些考慮將共同影響機(jī)器學(xué)習(xí)在下一代實(shí)驗(yàn)中的操作化。