Remove the peels from the citrus with a vegetable peeler a strip at a time, remove any of the bitter white pith by scraping off with the side of a knife.
Add all of the ingredients to a large pot, and simmer over low heat, covered, for about 30 minutes, stirring occasionally. Remove from the heat and pour through a strainer into a large bowl or glass pitcher. Add rum or brandy if using. Serve immediately or reheat to serve, or cool and chill in the refrigerator and serve cold. Each glass can be garnished with a cinnamon stick and apple slices if desired.