The ability to quantify the uncertainty in the prediction of a Bayesian deep learning model has significant practical implications — from more robust machine-learning based systems to more effective expert-in-the loop processes. While several general measures of model uncertainty exist, they are often intractable in practice when dealing with high dimensional data such as long sequences. Instead, researchers often resort to ad hoc approaches or to introducing independence assumptions to make computation tractable. We introduce a principled approach to estimate uncertaintyin high dimensions that circumvents these challenges, and demonstrate its benefits in de novo molecular design.