The scapegoat bit happened at the end of the liturgy of Yom Kippur. After the scapegoat was selected, there would be a crimson piece of wool tied to its horns, after offering all the other sacrifices, the high priest would place his hands on the head of a scapegoat, symbolically transferring the sins of the people onto the goat, and pray, "I beseech You, O Lord; Grant atonement for the sins, and for the iniquities and transgressions which the entire house of Israel has committed against You, As it is written in the Torah of Your servant, Moses: 'For on this day atonement shall be made for you, to purify you from all your sins - before the Lord you shall be purified'." The congregation responds with the words "Blessed be the Name of His glorious kingdom, for ever and ever."
Then, the scapegoat was taken out into the desert to a place called Azazel where the priest would push the goat off of a cliff. Before he did that, he would take a piece of the wool that had been tied around the goat's horns because once the goat had died, the wool would turn white fulfilling the scripture, "Though your sins are as scarlet, they shall whiten as snow; though they be red like crimson, they shall be white as wool" (Isaiah 1:18)
Hebrews 9:11-14 talks about Jesus being the replacement for the goats used on the Day of Atonement. Dying once and for all for the forgiveness of sin.
What is interesting to me about all this is that we still use this term to refer to people who are blamed for the wrongs of others. This happens often in families. Saying "If it wasn't for your ________ I/they wouldn't be so _________." The problem is that no other human can take the consequences of your sin for you. The only one that can do that is Jesus. He did it once and for all. He is the only scapegoat that will stand up under such a heavy burden. Maybe it's time for you to unload... don't worry, He can take it. He already did.