I'm currently using JR as a trial. I want to start out by saying this is by far the most top-notch extension in Joomla I've seen. Believe it or not, I think the admin interface may be easier to use than the joomla back end itself.
Thanks!
1) My review system is going to have thousands of cities, each with the same categories (such as restaurants, bars, pubs, etc). What is the best way to organize this using jR and joomla? From my understanding, if I make the cities the sections, then I'll have to recreate the categories for each city I create (doing so in the back end). Is there another way around this?
It's best to use a custom field for the categories and take advantage of the fields module and click2search functionality to allow users to filter by category. Then you use the same field for all cities.
2) Moreover, let's say on the home page I have a "restaurant tab." When someone clicks on this tab, I want them to be able to type in their city into a search bar, and then the top-rated restraunts in their city pops up. Is this possible?
You can use the advanced search module to let users pick a city. Currently search is not done on category names, so unless the city is repeated in content itself it wouldn't be found. You can set the default sorting in the configuration by rating.
3) How easy is it to modify jReviews? For example, some things I want to modify:
Entirely depends on your coding skills

Change stars rating system to A to F
You just need to change the function that generates the rating dropdown. (jreviews.common.php, selectSlider). Everything is changing in jReviews 2.0.
Make the overall rating itself a category for users to rate, not one that is averaged out by the criteria
You need to use a custom field for this, add the rating options individually. However, the average will still be calculated and it's the one used for sorting etc. You can get rid of it from the display side of things, but the functionality related to that average remains.
Other various changes to the layout of the reviews
Check the template forum. Again this will change in jReviews 2.0 providing greater flexibility.
Give the ability to filter results by "top-rated" for each criteria
Can't be done.