So there are a few suggestions that will make your life easier in terms of driving traffic to your site, they vary and it is really a case of doing whatever is easiest for you, it also depends on your site as to if there is much point in following some of the suggestions.
1. Invest or use a SEF component (Which you already have done, which SEF component?)
2. Sign up for an account with Google Analytics & Embed the code into your index.php file, this provides very useful information on referers to your site etc.
3. Create a XML for your website, some SEF components do this for you, or you may find some components on
Extensions.joomla.org which do this. The aim is to look at finding something that does XML sitemaps, and if you signup for Google Webmaster Tools there is a bit that lets you submit your XML sitemap URL to google, this gives google the links to all your pages and makes the crawling process quicker / easier, especially if you have alot of content.
Google works out relevance based on a thing called PageRank, which is a complex formula, but basically says that if website A has 10,000 other sites linking to it, and website B has 5 sites linking to it, website A would be more relevant given the same search terms, it also works this out based on the popularity of the sites linking, so having 5 unknown sites might not be as good as one well known (high pagerank site). But the point is you can also improve your ranking by posting in popular forums with your website link in your signature, by submitting news articles to Joomla news and other forums, and just spread the work about your site.
Good luck with the site

and the interesting process of SEO.