The Beastie Boys cut one album with Def Jam. It was their 1986 debut Licensed To Ill and it sold a cool ten million copies.
But the Beasties left the label after the album. According to Def Jam co-founder Russell Simmons, it was because of conflict between the band, producer (and Def Jam co-founder) Rick Rubin, and the Beasties' manager Lyor Cohen.
He also told Vlad TV that he thought the Beasties leaving Def Jam was a mistake, and if they had stayed Rubin's ability to make commercial albums which also stayed true to the artists' essence would have made their career even bigger than it eventually became.
Rick being the genius he was, I’m sure their next album would have been greater. I’m sure that had Rick stayed with the Beastie Boys they would have been Eminem or greater," Simmons said.That "next album" was 1989's Paul's Boutique. Although it was considered a commercial disappointment, it's long been a critical darling. (It also ended up selling 2 million.)
Rubin has done great things with everyone from Johnny Cash to The Red Hot Chili Peppers to Kanye West. So it would have been interesting to see what would have happened if he got another shot with the Beastie Boys.