## Abstract

Creating and controlling the topological properties of two-dimensional topological insulators is essential for spintronic device applications. Here, we report the successful growth of bismuth homostructure consisting of monolayer bismuthene and single-layer black phosphorus–like Bi (BP-Bi) on the HOPG surface. Combining scanning tunneling microscopy/spectroscopy with noncontact atomic force microscopy, moiré superstructures with twist angles in the bismuth homostructure and the modulation of topological edge states of bismuthene were observed and studied. First-principles calculations reproduced the moiré superlattice and indicated that the structure fluctuation is ascribed to the stacking modes between bismuthene and BP-Bi, which induce spatially distributed interface interactions in the bismuth homostructure. The modulation of topological edge states is directly related to the variation of interlayer interactions. Our results suggest a promising pathway to tailor the topological states through interfacial interactions.

## INTRODUCTION

Monoelemental two-dimensional (2D) materials with honeycomb lattice beyond graphene have been theoretically predicted to be ideal candidates of 2D topological insulators (TIs), in which a nontrivial topological gap is opened because of spin-orbit coupling (SOC) and topological edge states appear inside the gap (*1*–*4*). However, unlike predicted monoelemental 2D materials with freestanding form, the synthesis of these materials are usually supported by suitable substrates. In this case, the unavoidable strong interfacial interactions exert important effects on the 2D TIs and may break their nontrivial topological invariants. On the other hand, the interfacial interactions have been suggested to be a promising way to tune the topological electronic states of 2D layers (*5*–*7*). Recently, moiré superlattices formed because of the peculiar stacking sequence between neighboring layers in 2D heterojunctions represent a regular modulation of interlayer interactions and contribute an extra degree of freedom for manipulating the electronic structure of 2D materials (*8*–*18*). For example, twisted bilayer graphene at certain magic angles can host a flat band at the Fermi level to realize carbon-based heavily correlated electrons and even superconductivity (*15*–*17*) or provide AB-BA alternatively stacking domain boundaries to create the topological states (*12*, *13*). Triangular moiré potentials in graphene/hexagonal boron nitride (h-BN) heterostructure can clone the second replica of Dirac cones (*8*, *9*). However, despite the intense interests in the modulation of topological properties (*4*, *5*, *14*, *19*–*25*), experimental investigations and theoretical understanding of tunable topological electronic states in moiré superstructures of 2D TIs are still lacking.

Single-layer bismuth with honeycomb lattice, known as bismuthene, is regarded as a typical monoelemental 2D TI with the largest nontrivial gap (larger than 0.6 eV) due to the strong SOC (*2*, *5*, *26*). In practice, bismuthene has been synthesized on a number of different substrates (*27*–*30*). However, the strong bonding of Bi atoms to the substrates due to the lattice alignment between bismuthene and the underlying substrates hinders the observation of topological edge state tuning by interfacial interactions. Therefore, the moiré superlattice in the bismuthene heterostructure or homostructure provides a better system to investigate how the topological properties of 2D TIs are tuned by interlayer interactions.

In this work, we report the successful realization of bismuthene homostructure consisting of monolayer bismuthene and single-layer black phosphorus–like Bi (BP-Bi) on highly oriented pyrolytic graphite (HOPG) by molecular beam epitaxy (MBE). Various moiré superlattices with different twist angles between the two layers are observed. Scanning tunneling microscopy/scanning tunneling spectroscopy (STM/STS) investigations indicate the moiré superlattices accompanied by the modulation of topological edge states in bismuthene. Noncontact atomic force microscopy (AFM) measurements and first-principles calculations reveal that the tuning of topological edge states of bismuthene results from the regular variations of interlayer interaction in the moiré superlattices. Our work explicitly highlights that interfacial interactions can efficiently manipulate the topological properties of 2D materials, which is promising for 2D topological spintronic device applications.

## RESULTS

### Moiré superlattice formed in bismuthene/BP-Bi homostructure

Monolayer bismuth grown on most substrates have a black phosphorus–like puckered structure (BP-Bi), which can be regarded as an optimized bilayer of Bi(110) with both sides of dangling bonds totally self-saturated (*31*). As a consequence, unlike metallic bulk bismuth, monolayer BP-Bi with a thickness of 6.6 Å is predicted to be a 2D semiconductor (*32*). The top and bottom sublayers in the BP-Bi structure have a subtle buckling along the perpendicular direction (*31*–*33*), resulting in a rectangle lattice instead of homogeneous zigzag chains observed in STM/AFM images.

The growth of Bi atoms on HOPG at room temperature results in few-layer BP-Bi (*34*), but the situation is quite different for the growth of Bi atoms at lower substrate temperature (below ~220 K) in our experiment. After deposition, we found that the first layer of Bi on HOPG is a single-layer BP-Bi by STM (Fig. 1A) and AFM (Fig. 1C) measurements (section S1). However, the second layer of Bi exhibits a totally different atomic structure and thickness from the first layer. A honeycomb lattice is clearly resolved by high-resolution AFM imaging (Fig. 1D) and a height of 4.5 Å obtained from the AFM height measurements (fig. S1, C and D). We measured the lattice constants of first and second layer of Bi by statistical scanning along different lattice directions to reduce the uncertainties caused by tip drift and calibration errors, and the results are shown in Fig. 1E. The lattice constants of the first layer are determined to be *a* × *b* = 4.75 Å × 4.42 Å, which coincides well with the theoretically predicted value of freestanding BP-Bi (*21*), suggesting weak interactions between BP-Bi layer and HOPG surface. Nevertheless, the measured lattice constant of the second layer (4.20 Å) is close to freestanding bismuthene with honeycomb lattice (4.34 Å). Combined with the AFM observation (Fig. 1D), the second Bi layer can be assigned to be bismuthene with buckled honeycomb lattice. The small deviation of lattice constant (−3%) from freestanding form indicates compressive strain induced by interlayer interactions in the bismuthene/BP-Bi homostructure, which was also observed in graphene/h-BN structure (*35*).

Two layers stacking together with different lattice constants and symmetries usually results in a moiré pattern, which is observed for the bismuthene/BP-Bi homostructure (Fig. 1A). The statistically measured distribution of twist angle α between bismuthene and BP-Bi is shown in Fig. 1F, indicating that α’s around ±2.0° are more frequently observed, and the areas of homostructures with these angles are also much larger than the others. Therefore, the moiré superlattices with such twist angles of ±2.0° are more stable, and the following investigations focus on such systems.

In principle, the structure of moiré superlattice is determined directly by the lattice constant of sublayer and overlayer, as well as their twist angle. Combining the commensurate superlattice configuration (*36*) with our experimental observations, the atomic model of bismuthene/BP-Bi homostructure with twist angle of −2° is constructed and shown in Fig. 2A (section S3). Because of the small lattice mismatch and minor twist angle, there are three large inequivalent stacking areas in the superlattice unit cell: top, bridge, and face-centered cubic (fcc) areas. These areas can be distinguished by the Bi atoms in bismuthene layer located at the top, bridge sites of BP-Bi layer, and the Bi atoms of BP-Bi layer appearing at the hollow sites of bismuthene, respectively.

This moiré superlattice model is further investigated by density functional theory (DFT) calculations. After structural optimization, the lateral positions of Bi atoms are almost unchanged. However, the calculated interlayer distance modulation and out-of-plane surface fluctuation are both found in the moiré superlattice of homostructure. The interlayer distances between bismuthene and BP-Bi layers are calculated to be 3.55, 3.45, and 3.11 Å in top, bridge, and fcc areas, respectively, with a fluctuation as large as 0.39 Å (Fig. 2I), i.e., the interlayer distance in the fcc areas is much shorter than that for top and bridge (T&B) regions. This means that the interlayer distance between bismuthene and BP-Bi layers varies gradually in regions with different stacking modes (Fig. 2B). Consequently, a stripe-like fluctuation on the surface of bismuthene occurs, which is verified by our AFM measurements as shown in Fig. 2E. The simulated STM image of the moiré superlattice model produces the same alternating bright and dark strips along the long axis of the superlattice (Fig. 2, C and F, and fig. S4).

Experimentally, we performed force spectroscopic measurements by AFM to accurately determine the height differences between surface Bi atoms in the moiré superlattice. On the basis of single element (same van der Waals radius and chemical force) and conducting property of surface bismuthene (homogeneous electrostatic force), the distance-dependent atomic force of Bi atoms at different locations in bismuthene should remain identical. The turning point of force spectroscopy at a specific tip-Bi distance represent the height of each Bi atoms in the bismuthene (*37*, *38*). As shown in Fig. 2 (G and H), force spectroscopies of Bi atoms in the three inequivalent areas exhibit the distance shift along *Z* direction, suggesting the different height of Bi atoms in the three inequivalent areas. Quantitatively, the height differences of 0.11 Å between T&B sites and 0.39 Å between top and fcc sites are obtained. The height measurements along two different lattice directions illustrate the continuous variation of surface height (fig. S5). Consequently, the interlayer distance variations in moiré superlattice of bismuthene/BP-Bi homostructure are confirmed by the good match between experimental results and theoretical calculations (Fig. 2I).

### Topological state modulation by moiré superlattice

Monolayer bismuth with honeycomb lattice (bismuthene) has been theoretically and experimentally verified to be a prototypical 2D TI (*2*, *5*, *27*, *30*), in which nontrivial band topology is robust against strain, electrical field, and substrate interactions (*5*, *39*). To interrogate the electronic structures, we performed STS measurements on the bismuthene/BP-Bi homostructure. As there are only zigzag edges in bismuthene single layer on the BP-Bi surface (fig. S6), the hexagonal bismuthene islands with only six straight zigzag edges are preferable for experimental measurement. Figure 3A shows the *dI*/*dV* spectra taken at three inequivalent stacking areas (top, bridge, and fcc) of the moiré superlattice at edge and center part (bulk) of a bismuthene island. An obvious state around 100 meV above the Fermi level is observed at the edge. Figure 3B is the line mapping of *dI*/*dV* along the T&B strip from the center (0 Å) to edge (89 Å) part of the bismuthene island (Fig. 3C), which indicates that this state is located mainly at the edges of the island with a penetration depth of 28 Å. We do not observe any structural reconstruction at the edges, and the state is immune to the defects at edges (section S4). This experimentally observed state is proposed to be the nontrivial topological edge state in bismuthene.

The edge states at fcc areas of the moiré superlattice are strongly suppressed compared with T&B areas (Fig. 3A). To further confirm this, we spatially resolved the edge states in the *dI*/*dV* map and compared with corresponding STM morphology (Fig. 3, D and E). It is clear that the edge states have the correlated spatial distribution with the moiré superlattice. That is, the edge states at T&B areas along the long axis of the superlattice are much more robust than those at fcc areas. Moreover, the line profiles (Fig. 3F) along the T&B strip and fcc strip indicate that the density of edge states (at 100 meV) at fcc areas is much weaker but has similar spatial distribution as that at T&B areas, suggesting that the edge states are still retained but just suppressed at fcc areas. From the semiquantitative measurements of differential conductance at the three different sites (Fig. 3G), the modulation of edge states related to the moiré superlattice is also illustrated. The density of states (DOS) at T&B sites are comparable to each other but higher than fcc, which correlates with the modulation of interface distance in moiré superlattice (Fig. 2I).

To understand the modulation effect of the moiré superlattice on the topological edge states in bismuthene/BP-Bi homostructure, we performed DFT calculations based on two simplified bismuthene/BP-Bi nanoribbon structures, as shown in Fig. 4 (A and B, respectively). The interlayer distance between bismuthene and BP-Bi is fixed to the calculated interlayer values (3.5 Å in T&B nanoribbon and 3.1 Å in fcc nanoribbon) to account for the different interlayer interactions. To simplify the calculations, the twist angle is set to zero and the lattice constant of bismuthene is set to match BP-Bi (4.42 Å; section S5). As a result, T&B and fcc stacking areas appear alternatively along the direction perpendicular to the nanoribbon long axis. Specific edges terminated by T&B and fcc stacking can be obtained by adjusting the relative position of bismuthene and BP-Bi. For each nanoribbon, there are two different atomic endings: ending with upper buckled atom (type A) or with lower buckled atom (type B). Consequently, there are four kinds of edges in calculated nanoribbons, i.e., T&B-A, T&B-B, Fcc-A, and Fcc-B, as highlighted in Fig. 4 (A and B).

The calculated band structures of the two nanoribbons with four edge endings are shown in Fig. 4 (C and D). The edge states can be clearly distinguished similar to the freestanding form (fig. S8C). In T&B nanoribbon, both edge states of edges T&B-A (blue) and T&B-B (red) connect conduction band and valance band, which supports the nontrivial topology of T&B-stacking bismuthene. However, in fcc nanoribbon, edge states are repulsed into either conduction band or valance band near Γ point because of the stronger interlayer hybridization, resulting in a trivial electronic state. Further detailed interlayer interaction can be closely inspected in the calculations of charge density distribution in Fig. 4 (E and F). The same as the published work (*40*), the local DOS distribution of edge states around 100 meV for type A and type B edges in both T&B and fcc nanoribbons are different. As the outmost lower buckled Bi atoms in type B edge are much closer to the underlying BP-Bi atoms, the electronic states at type B edge overlap with BP-Bi layer strongly, suppressing the topological property of bismuthene. Similarly, here, we also find that the coupling between Fcc-A edge and BP-Bi is stronger than the T&B-A edge, indicated by the severe penetration of edge states into BP-Bi, as shown in Fig. 4F. Therefore, the stronger delocalization at fcc area correspondingly explains the observed edge state suppression in our experiment.

## DISCUSSION

Previous research has suggested that the quantum properties of 2D materials, for example, the topological property in 2D TIs, are controlled by several factors (*4*, *5*, *20*–*23*). Among them, substrate (interlayer) interaction is one of the most efficient and powerful means to tune the quantum properties. For few-layer bismuth with honeycomb lattice, the *Z*_{2} numbers were theoretically predicted to be tunable by interlayer distance or the number of layers (*5*, *41*). For instance, increasing the interlayer distance between two Bi(111) layers to over 4.32 Å will convert it from a nontrivial to trivial insulator (*5*). In the bismuthene/BP-Bi homostructure, the interlayer distance (3.11 to 3.55 Å) is larger than bulk Bi(111) interlayer distance of 2.32 Å, suggesting that the formation of moiré superlattice is indicative of an electronically decoupled bismuthene layer from BP-Bi. Therefore, instead of delocalization of edge states due to the strong hybridization between type B zigzag edge and substrate in the Bi(111) terrace edge (*40*, *42*), both type A and type B zigzag edges in our bismuthene/BP-Bi homostructure preserve their topological edge states, as observed in the experiments (Fig. 3E) and calculations (Fig. 4C). In the moiré superlattice, the interlayer distance decreases from top to fcc stacking areas, with gradually increasing interlayer interactions. As a consequence, the stronger coupling at fcc stripe areas will induce the suppression or breaking of topological edge states, resulting in the experimentally observed striped modulation of topological edge states.

Epitaxial growth of 2D materials on substrates ordinarily requires symmetry matching for both thin film and substrate. However, monoelemental 2D materials with buckling honeycomb structures, such as monolayer Si (*43*), Ge (*44*), Sn (*45*), Sb (*46*), and Bi (*27*), are likely to register with substrate lattice, which brings in strong interface interactions and ordinarily breaks the intrinsic topological properties by strong orbital hybridization with substrate. In this circumstance, a moiré superlattice with twist angle would be an alternative means to reduce the substrate interactions due to lattice incommensurability between 2D material and substrate. With different symmetry, the formation of hexagonal bismuthene on rectangular BP-Bi in this work may offer a new paradigm for decoupled monoelemental 2D TIs to realize the quantum spin Hall effect.

## MATERIALS AND METHODS

### Sample preparation and characterization

Experiments were performed in an Omicron LT-STM system equipped with an MBE chamber (1 × 10^{−10} mbar). HOPG was cleaved in air and then loaded into MBE to degas to 800 K. Bismuth (99.999%) was deposited on the HOPG substrate for 5 min, and then, the sample was in situ transferred to an STM chamber immediately. All the STM/STS and AFM measurements were carried out at 77 or 4.3 K (data measured at 4.3 K are indicated in the captions) with a tungsten tip on qPlus AFM sensors. Clean and atomic sharp tip was obtained by repeat-voltage pulsing and controllable poking on the Au(111) surface. STS was obtained by a lock-in technique, with which a 20-mV (root mean square) and 963-Hz modulation was superimposed on the sample bias. For AFM measurements, we used an ordinary metal tip with oscillation amplitude of 3.00 Å at 77 K and 1.00 Å at 4.3 K to record the frequency shift (Δ*f*). All the STM/AFM images have been processed using WSxM software (*47*).

### First-principles calculations

First-principles calculations of the geometric and electronic structure of bismuthene superlattice and its simplified nanoribbons were performed in the framework of DFT using the Vienna Ab Initio Simulation Package with a plane wave basis (*48*). The generalized gradient approximation function of Perdew-Burke-Ernzerhof and the projected augmented wave pseudopotential for the ionic cores and valence electrons are used (*49*, *50*). The plane wave basis set cutoff for wave functions is set to 350 eV. Aiming to mimic the experimentally observed superlattice, we build a 216-atom supercell composed of single BP-Bi and single buckled honeycomb Bi slab with in-plane lattice constant to be 44.56 Å × 16.81 Å. Considering the large computational costs, the bottom layer atoms of BP-Bi are fixed, and only the top three layers are fully relaxed during structural relaxation until the force is less than 0.02 eV/Å. The supercell has a vacuum layer more than 15 Å perpendicular to the slabs. To investigate edges states of the bismuthene film, similar calculations are carried out for 1D bismuthene zigzag nanoribbon on BP-Bi thin film with a width of 15 zigzag chains, and the in-plane vacuum layer is at least 20 Å to ensure decoupling between neighboring ribbons. Interlayer distance is specified in different nanoribbon to investigate distance-dependent electronic properties. The 2 × 4 × 1 γ-centered *k*-mesh is used for periodic superlattice of bismuthene homostructure, while the 9 × 1 × 1 mesh is used for bismuthene nanoribbon on periodic BP-Bi surface. SOC is only included in the calculation of electronic structure of nanoribbon. Simulated STM image is calculated using Tersoff-Hamann approximation (*51*, *52*).

This is an open-access article distributed under the terms of the Creative Commons Attribution-NonCommercial license, which permits use, distribution, and reproduction in any medium, so long as the resultant use is **not** for commercial advantage and provided the original work is properly cited.

**Acknowledgments: **We thank J. Lu and J. Su for fruitful discussions. **Funding:** This work was supported by the NRF of Singapore (grant no. R-144-000-405-281), the MOST of China (grant nos. 2018YFE0202700 and 2016YFA0202301), the NSF of China (grant nos. 11761141013, 11674366, and 11974045), the Beijing NSF (grant no. Z180007), and the Strategic Priority Research Program of the Chinese Academy of Sciences (grant nos. XDB30103000 and XDB30000000). **Author contributions:** J.G., L.C., and A.T.S.W. proposed and conceived this project. J.G. performed the MBE preparation and SPM measurement under the advice of L.C. A.T.S.W. and L.K. did the calculations under the advice of J.S., S.M., and K.W. L.C., X.H., and Y.L.H. participated in the experimental measurement. All the authors contributed to the data discussion and manuscript preparation. **Competing interests:** The authors declare that they have no competing interests. **Data and materials availability:** All data needed to evaluate the conclusions in the paper are present in the paper and/or the Supplementary Materials. Additional data related to this paper may be requested from the authors.

- Copyright © 2020 The Authors, some rights reserved; exclusive licensee American Association for the Advancement of Science. No claim to original U.S. Government Works. Distributed under a Creative Commons Attribution NonCommercial License 4.0 (CC BY-NC).