Boil a large pot of salted water. Add cauliflower and cook until very tender, about 10 minutes. Drain.
Meanwhile, in a separate saute pan, warm olive oil over medium-low heat, add garlic and saute for about 30 seconds, or until the garlic is golden and fragrant. Remove from the heat.
Mash with a potato masher in a large bowl. For a smoother mashed cauliflower, use an immersion or hand blender to puree, or puree it in the food processor. While mixing, slowly add milk until the desired consistency is achieved. Mix in the salt and pepper and the minced garlic with the olive oil. Serve.
Mashed cauliflower can be topped with chopped scallions if desired.