One reason was that some workers feared that if slaves were freed, they would come North and compete with them for jobs. They worried that this competition would drive down wages and make it harder for them to support their families. Northern workers sometimes rioted over the competition they faced from recently arrived immigrants, and they might fear that freed slaves would face similar prejudice and aggression.
Another reason was that some workers believed that slavery was necessary for the economy. They thought that the southern economy would collapse if slavery were abolished, and that this would have a negative impact on the national economy. They were concerned about losing their own jobs if the country went into an economic recession as a result of the end of slavery.
Third, some workers didn't feel that the issue of slavery affected their lives directly. Especially in the 1830s, there were several groups of "free labor" advocates within the labor movement who took anti-slavery stances on the grounds of economic or philosophical principles. These "free labor" groups were most prominent among artisans in New York City, the largest metropolitan center in the United States at the time. However, the broader labor movement was often less involved in anti-slavery activism.
Of course, it's important to remember that not all northern workers opposed the abolition of slavery. Many white workers in the North supported the abolitionist movement and worked to end slavery.