Gospel Guidebook: Getting and Keeping It Right 
Evidence and Comments on Penal Substitutionary Atonement
The Evidence
My understanding is that there are six explicit passages in the New Testament that teach substitution. There are three passages with the preposition anti (Matthew 20:28, Mark 10:45, and 1 Timothy 2:6) and three with the preposition hyper (John 11:50, 2 Corinthians 5:14, and Galatians 3:13).
We also know that the early church regarded hyper as carrying a substitutionary force similar to anti. For example, we have a writing from Irenaeus (2nd century) in Against Heresies Book 5, Chapter 1 where he says the following.
"Since the Lord thus has redeemed us through His own blood, giving His soul for (hyper) our souls, and His flesh for (anti) our flesh"
τῷ ἰδίῳ αἵματι λυτρωσαμένου ἡμᾶς τοῦ κυρίου καὶ δόντος τὴν ψυχὴν ὑπὲρ (hyper) τῶν ἡμετέρων ψυχῶν καὶ τὴν σάρκα τὴν ἑαυτοῦ ἀντὶ (anti) τῶν ἡμετέρων σαρκῶν
In the Greek, we can see that he uses the prepositions hyper and anti in the exact same way, carrying substitutionary force.
In addition to Irenaeus, we have the research from A. T. Robertson and Daniel Wallace that show 87 examples from early papyri, a few from the 1st century and some from the 2nd century, that show idiomatic and non-idiomatic use of hyper as bearing substitutionary force. In addition to them, J. U. Powell found a couple hundred examples in the Oxyrhynchus Papyri. (See Wallace's Beyond the Basics pages 383-389.)
So, in light of the evidence of hyper bearing substitutionary force, we have not only those six explicit passages that I mentioned above, but we also have many more, granted that hyper was interpreted in the sense that Irenaeus did. The very fact that hyper could be interpreted as substitutionary is in itself significant because this means that the Apostle Paul, etc. would have known that his readers (at least some of them) would have probably interpreted hyper as bearing substitutionary force.
And besides the actual Greek texts and the perception that these texts give, there are motifs of substitution throughout the whole Bible, including the sacrifices and Barabbas at the cross.
Comments on Substitution
I believe in substitutionary atonement, and I believe the evidence for it in terms of the Scripture verses provided above, the motifs, and perception given by the Old Testament sacrifices, is substantial and conclusive beyond any reasonable doubt. Furthermore, the gospel of the free grace of God seems to depend on it. If substitution is not true, it would be unclear how God could justify "the ungodly" as "being without strength," as "being sinners," as "being enemies of God," and as "being dead in trespasses" (all present participles) (Romans 4:5, 5:6, 5:8, 5:10 and Ephesians 2:5). God doesn't justify us because of what He does in us to change and transform us into moral people; rather, He justifies us solely as "ungodly people" on the basis of what Christ did for us in His life and death on the cross. It is hard to imagine how this could be possible if substitution and imputation were not true.
Comments on Penal Substitution
The concept of penal substitution bothers a lot people. However, to understand this, I really think we have to start with the important presupposition that God and His law are inseparable. (For people who reject this presupposition, I'd like to know why the ark of the covenant, which contained the commandments, acted as the foundation for the mercy seat, i.e., God's throne, if the law was in fact separable from God?) Given this presupposition, we need to consider that there can be no effective law without penalties. Now, God is not only the lawgiver, but also aggrieved party when His law is broken. The Apostle Paul tells us, "The law worketh wrath" (Romans 4:15). The cross dealt with both the penalty and the wrath, disannulled the old legalistic age, and initiated this new gracious age where a "righteousness of God apart from law" is available to all people who believe in Christ (Romans 3:21-22). More specifically, since the cross marked the closure of the old age, the debts of the old age needed to be settled. Christ settled all debts by performing "redemption of the transgressions that were under the first testament" (Hebrews 9:15). In this way, we can see that the cross had a legalistic aspect to it. For more details, I recommend taking a look at the article The Wrath Of God In The Old Testament: "The Law Brings Wrath" (offsite link).
Deliverance from wrath finds its basis in the atonement and is the result of the atonement, for Paul tells us, "Much more then, being now justified by his blood, we shall be saved from wrath through him" (Romans 5:10). Atonement relates to making peace with God. An important word that relates to the atonement is hilasterion, in Romans 3:25, which is often translated as "propitiation." A few Bible translations, such as the Christian Standard Bible and the NET Bible have opted to translate it as "mercy seat," drastically changing our perception of the word. While that translation seems appropriate for Hebrews 9:5 where we see the cherubim overshadowing the "mercy seat" (i.e., the place of votive offering), it doesn't really work well for Romans 3:25 where it is not the place of votive offering that is being emphasized but the actual votive offering itself. In this regard, A. T. Robertson in his commentary on Romans 3:25 cites evidence provided by Deissmann that "propitiation" is the correct translation, concluding that "there is no longer room for doubting its meaning in Romans 3:25." In addition, the burnt offerings and incense in the Old Covenant do seem to symbolize appeasement of wrath. In 2 Chronicles 29:6-8, we learn that one of the reasons why God acted with wrath against Judah was because she had forsaken the burnt offerings and incense. Their sinful behavior created the wrath, and their forsaking of the burnt offerings and incense left them unable to appease the wrath. Likewise, in Numbers 16:46–48, Aaron burned incense unto to God to make atonement and turn away His wrath.
I don't think I have even scratched the surface of the topic of substitutionary atonement, but the evidence for it is substantial enough to accept it as truth. Surely, there is much about it that we cannot understand, but this should not dissuade us from believing it as scriptural.