I realized that a Vim plugin called gutentags was creating huge tags files in my Git repos. In the gutentags docs I noticed a way to make it easier to manage all the tags files in various repos.
This line in .vimrc (after installing gutentags) tells the plugin to save all the tags in a specific directory.
let g:gutentags_cache_dir = "~/tags"
That should make it easier to delete them all at once without worrying about typos that could rm -f other files in my project folders.
I also got the swp files out of my projects by creating a directory called ~/.vimswap and putting this in .vimrc:
set dir=~/.vimswap//,/var/tmp//,/tmp//,.
Then the swp files end up in ~/.vimswap instead of cluttering up the project.
Seen here.