Existing model uses structured data to predict the patients of either high risk or low risk. But for a complex disease, structured data is not a good way to describe the disease. We propose a new convolutional neural network (CNN)-based multimodal disease risk prediction algorithm using structured and unstructured data from hospital. In this paper, we mainly focus on the risk prediction of cerebral infarction.