Miller later settled in Bungay, Suffolk, where the demand for books was small. Moreover, his sturdy independence lost him to the custom of many of the local magnates. As his son William recorded in a memoir:
Thomas Miller was little suited by nature to the quiet, unambitious life of a country shopkeeper. He possessed a strong & vigorous understanding, with great energy & industry. His mind, which he had improved by much reading, was adapted to a larger sphere - his cramped locality marred his fortune...He had a spirit too independent for the petty aristocracy that surrounds most country towns - his politics never varied or were adapted to his local interests....He had a talent for light poetry but was too fond of lampooning his neighbours when the vein was upon him & he saw a fair opportunity.
Miller's stock of books was very valuable, and he had an extensive collection of engraved portraits, and nearly a complete series of Roman and English silver and brass coins. He published catalogues of his collections in 1782 and 1790. In 1795, when the fashion was very general for tradesmen to circulate provincial halfpennies, he had a die-cast, but an accident happened to one of the blocks when only twenty-three pieces had been struck off, Miller declined having a fresh one made. This coin, finely engraved, with a strong profile likeness of Miller, is known to collectors by the name of the "Miller halfpenny".