In the 1860’s, droughts in California destroyed most cattle empires. Don Raymundo was able to save his empire by raising sheep instead of cattle.
In 1864, Don Raymundo’s business partner Felipe Lorenzana sold his half of Rancho San Miguel.
The death of Don Raymundo in 1879 was the beginning of the end for the Olivas' fortune. Though some family members retained pieces of the land grant until as late as 1968, the house was sold in 1899. Rebecca Olivas, youngest daughter of Don Raymundo, was the last member of the Olivas family to live at the adobe.