Microbial elevational diversity patterns have been extensively studied, but their shaping mechanisms remain to be explored. Here, we examined soil bacterial and fungal diversity and community compositions across a 3.4 km elevational gradient (consists of five elevations) on Mt. Kilimanjaro located in East Africa. Bacteria and fungi had different diversity patterns across this extensive mountain gradient—bacterial diversity had a U shaped pattern while fungal diversity monotonically decreased. Random forest analysis revealed that pH (12.61% importance) was the most important factor affecting bacterial diversity, whereas mean annual temperature (9.84% importance) had the largest impact on fungal diversity, which was consistent with results obtained from mixed-effects model. Meanwhile, the diversity patterns and drivers of those diversity patterns differ among taxonomic groups (phyla/classes) within bacterial or fungal communities. Taken together, our study demonstrated that bacterial and fungal diversity and community composition responded differently to climate and edaphic properties along an extensive mountain gradient, and suggests that the elevational diversity patterns across microbial groups are determined by distinct environmental variables. These findings enhanced our understanding of the formation and maintenance of microbial diversity along elevation, as well as microbial responses to climate change in montane ecosystems. © 2020 Society for Applied Microbiology and John Wiley & Sons Ltd.