The community of Partridge Island had lost it's most important resident. James Ratchford, Esq through death. And with him went the heart and soul of the small original settlement. Moves had already begun to shift from Partridge Island to Mill Village, the future site of Parrsboro.

The reasons for this shift become obvious with the perspective of time. The death of Ratchford himself would be one factor, and indicative of the shift is the fact that he was laid to rest in the Anglican cemetary in the 'new' town. His family attempted to carry on business as usual at Partridge Island, however it slowly declined in the general economic depression of the early 1840's. In addition, much of the land in the area remained for many years in family hands and were thus not available for community expansion.

When Josiah Davison with his sons Cyprian, John, Vickery, Aline and Edward established a highly successful grist mill on the tidal creek at Lower Victoria Street, the shift was almost inevitable. Farmers from all along the Parrsborough shore, even as far down shore as Advocate Harbour and up shore as far as Economy, travelled to this mill in boats to bring their grain. Even farmers from Cornwallis brought their wheat, corn and oats by boat to have them ground. The Davison mill even had a kiln with which to dry the oats and it was reported that they were able to manufacture it into the best oatmeal. "The flat rolled oats that is imported here at present can not be compared to it."

At the same time the large grant holders in the Mill Village area such as the Anglican Church, the Vickery's, the Davision's etc., were subdividing their lands. The site of Mill Village proved much more suitable for building sites than did the volcanic stone hills and shoreline of the Partridge Island community. Also the fine, protected harbour allowed for safer docking facilities and ship building sites.