The answer: none of those things, or at least not necessarily. The Federal Highway Administration has given the “control cities” listed on these mileage signs a “center point.” The criteria for a center point: whatever the local authorities want it to be. It could literally be anything, so long as the FHA approves it as a “well-defined central area.” That could be a downtown, a central business district, or city hall, for example. Mileage signs into Baltimore denote how far away City Hall is. In New York City, the distance measures to the busy Columbus Circle area in Manhattan.
Source: Do Geese Get Goose Bumps by Bathroom Reader’s Institute