Everyone has a different approach for books. Some use strict rules or charts of rank versus return, others tend to use experience and instinct. Or others use a combination smile emoticon
First let's talk price/return:
There are some general guidelines - first of all, value your time.
A book that sells for $9.99 will give you a payout of approx $5.20. $8.99 will get you a payout of about $4.35 . $7.99 will get you about $3.50. (It does vary though so use the FBA calc to check)
Be aware of your inbound shipping costs. Depending on your warehouse, a medium book can cost around 50c to ship to Amazon.
Figure out how long it takes you to list a book. I can probably list 60 books in an hour if I'm caffeinated and focusing. If I make $2-$4 on each of those, that's $120-$240 so selling 60 well-ranked but low return books (in the $7.99-$9.99 range) might be worthwhile. If it takes you more than a couple of minutes to list books, then it might not be worth your time to list low value books.
(Of course you have to factor in the price of the book too - I'm a fan of buying books cheap from yard sales or in bulk from library sales)
Let's talk rank.
First thing - Rank is SUPER misleading. Rank can jump from 2 million to 300,000 with one sale. I'm a big fan of camelcamelcamel.com which you can use to get a much clearer view of how many copies of that book have sold.
I don't have a limit on rank. Used books cost hardly anything to store at Amazon and I've sold books in the 8-13 million range at decent prices before. Yes it may take a year or two for them to sell but if it's a 50c investment and you get a payout of $50 in a year, then why not do it? smile emoticon The exception to this is if there is a very good reason the book won't sell - an outdated tax manual or something. But it still might be worth a gamble. If you have books in the 2-4 million range with a decent return ($15+) then I would absolutely send them in.
Anything under 1 million is likely going to sell within a month or two.
As far as what books to sell - anything that isn't a penny book is a good start - but developing the skill to spot valuable books at an estate sale or yard sale is the way to go. I remember when I first started researching book buying I was totally overwhelmed with volume and felt I could never learn to spot a great book. Now I have a general feel for where to go at a sale. Use elimination and ditch ANY bestsellers, Oprah's bookclub, Dr Phil, National Geographic or anything else produced for the mass market. This can get quite counterintuitive - you're generally looking for non-fiction (although special edition fiction can do well!) and obscure. So for example 'Academic Insights on Performance Studies in the United Kingdom' is probably going to be more valuable than 'Acting for Dummies'
Obscure, niche, academic texts are really where it's at. My biggest winners have been things like Gemstone faceting (teeny pamphlet), equine biology, essays about neonatal brain surgery, Japanese woodblock carving etc.
Recent tech books can be a goldmine too - programming, operating systems etc but if they're older than a couple of years, they might be duds. You can usually get a feel for them quickly by scanning a few.
With books, the more you find and sell the more you get to feel what works.
This video might explain a bit more about camel and sales rank :)