In the case of virtually all practical applications, classification algorithms are required to construct models that minimize a non-uniform loss function, rather than the 0/1 loss. One of the most efficient approaches to do this is to first estimate the class probabilities of the unseen instances and then to make the decision based on both the computed probabilities and the loss function. Learning models that compute accurate class probability estimates is - in general - known to be a difficult task. As a result, large research efforts are made to improve the accuracy of the estimates computed by different algorithms. This paper presents a novel approach to learning classification models for making cost-sensitive decisions by addressing the problem of minimizing the actual loss associated with the decisions rather than improving the overall quality of the probability estimates. Our approach relies on employing ensembles for estimating confidences for the learned class probabilities. The classification decisions rely on the loss function and the position of the decision boundary with respect to the estimated confidence interval. For the experimental analysis we have implemented our methods using different types of ensemble algorithms: bagging, random trees, and random forests. The confidence intervals for the probability estimates are computed based (1) on counts and (2) on the normal approximation of the estimations of the base classifiers learned by the ensemble. The results show that for some tasks, the proposed algorithms outperform some of the best probability estimation-based algorithms for cost-sensitive classification. Further analysis provides other interesting insights into learning models for cost-sensitive decision making.