Add olive oil to cast iron skillet (or a regular frying pan is OK) over medium-high heat. Once oil is hot enough, drop 1/4 cup of fritter batter into the pan and flatten out with a spatula. Depending on the size of your skillet, cook a few fritters at a time, leaving space in-between. Cook for a 3-5 minutes on each side, or until golden-brown and crispy.