As you build up your garrison and advance through its levels, it becomes possible to make a lot of different choices about what kind of buildings to construct. Each level gives you certain sized plots to place your buildings on, and each building has different requirements for what kind of work it can do. Luckily, if you decide that one of your level 1 buildings isn't really doing what you wanted it to do (say the Barracks for your soldiers), you can delete it and replace it with something else.
The first time you build your garrison, it's likely that you'll be asked to create the Barracks, which is fine -- this will unlock patrol missions for your followers to complete, which will give them a lot of XP and help them get to epic levels very quickly. There are also special buildings that can be constructed once you reach level 2, the Mine, Fishing Shack, Herb Garden and Menagerie. These buildings will each offer a unique kind of functionality for your garrison, allowing you to do things like craft new recipes or harvest particular items.
One of the problems with these buildings, though, is that they can cost quite a bit of resources to upgrade and then maintain at each level. In order to do so, they must be constantly occupied by followers, who will run missions and gather up resources for the building. Using a resource calculator can show you just how much of a financial commitment a single building is, and if you have a lot of these buildings that aren't being used, it might be better to delete them in favor of something more useful.