In the days of the general store, cloth came in bulk and was sold by the yard.
The storekeeper, who quickly became expert at measuring, often used the length of his arm as a measure of each yard being purchased.
If the measurement was challenged, the seller would remeasure the cloth against two brass tacks embedded in the counter that were precisely a yard apart.
The issue was therefore settled by getting down to those two brass tacks.
The phrase is commonly used when referring to getting down to serious business.