Put a cast-iron or other heavy skillet over medium-high heat. Pour in the oil, add the rosemary, and stir to infuse the flavor into the oil. Heat the oil to around 300°F. (You don’t need a thermometer for this: Drop an almond into the oil. When it sinks a bit, then floats right back to the top and starts sizzling, the oil is hot enough.) Once the oil is ready to go, add half of the almonds and cook, stirring constantly, until they are fragrant and toasted, 10 to 20 seconds. Using a slotted spoon, remove the almonds from the oil and drain on paper towels. Sprinkle them with the salt while they are hot. Repeat with the remaining almonds. Use to top the marinated olives or serve as a snack.