The purpose of this paper is to examine the main security problems in electronic voting systems, particularly security threats to DRE voting systems and security threats to the Internet voting systems. It will focus on how security problems can be addressed. The paper is divided into four parts. The first part will pinpoint the criteria of using electronic voting systems while focusing on the main security problems in DRE and Internet based voting systems and will suggest their solutions. The second and third parts will propose secure reference architecture for electronic and internet based voting systems while the last part will be the conclusion.