Urban agriculture has increased in many cities and has the potential to provide an array of benefits including increased local food production, nutrient recycling, urban green space, and biodiversity. While certain environmental benefits of urban agriculture are evident, it is not clear what the optimal extent of urban agriculture would be in designing a sustainable city. Closing the loop by recycling waste products into new resources is fundamental to sustainability, but the extent to which this should occur at local, regional, or global scales is an open question. We analyze how potential benefits and costs associated with urban agriculture scale with the extent of implementation, and compare potential tradeoffs in different metrics of sustainability. We assess how the appropriate metrics to optimize in a given city are context-dependent. For example, maximizing production in a small land footprint could be important in densely developed urban environments, whereas filling vacant land with food-producing gardens may be a more appropriate goal in certain post-industrial cities. We assess the potential role that urban agriculture plays in making urban food systems more resilient to climate change and other disruptions. Finally, we consider a case study comparing the resources required and pollution generated to produce the lettuce supply of on U.S. metropolitan area through outdoor urban agriculture and indoor urban agriculture, compared to conventional production and cross-continental transportation. This analysis illustrates the importance of considering multiple metrics in assessing sustainability of urban agriculture.