Your French dictionary will define the expression hors d’oeuvres, “in composition, art, or the like, that which is not an essential part of the work.”
Then it may go on to say, “In cuisine, something such as radishes, olives, etc., served before the main dish.”
From this you may see that the literal idea of the phrase is, “Aside from the work; not essential.”
