To determine the number of genomes within the population automatically could use F-statistics (http://en.wikipedia.org/wiki/F-statistics). It doesn’t seem like it has been explored in genetic algorithms, rather for human population studies. Another way that subpopulations could be culled is to use the inbreeding coefficient (http://en.wikipedia.org/wiki/Coefficient_of_relationship).