Swansea is bounded on the west by the Humber River, on the north by Bloor Street, on the east by High Park and on the south by Lake Ontario. The neighbourhood was originally a separate municipality, the Village of Swansea, which was annexed by the City of Toronto in 1967.

Swansea is primarily residential in nature, consisting of a mix of various housing types. Swansea's high-end homes are located either at the western edge of High Park overlooking Grenadier Pond, or on Riverside Drive and the Brule Gardens enclave bordering the Humber river. Swansea also contains a large number of semi-detached houses and bungalows located mostly in the centre of the neighbourhood. The typical house of the area was built between 1905 and 1935. 

The area of the former Swansea Works area is considerably newer, except for some of the original workers' homes. The Queensway was built in the 1950s through the Swansea Works lands. The area to the south was retained for industry and the area to the north was redeveloped with apartment buildings and townhome developments. The area of the actual factory site has been redeveloped since 2000 into townhomes and condominium apartments.

Swansea has several main streets. Along the northern boundary, Bloor Street is a four-lane arterial road with businesses lining both sides. Along the southern boundary, The Queensway is a four-lane arterial road with a streetcar right-of-way. The Queensway has primarily residences on both sides. North-south, Swansea has two major roads, South Kingsway and Windermere Avenue. Along the southern boundary is the Gardiner Expressway which has an interchange with South Kingsway and the CNR railway lines. Further to the south, Lake Shore Boulevard runs east-west parallel to the lake shore.

The area is extremely hilly in nature. The waters of Grenadier Pond, Rennie Pond and the Humber River all are at or near the level of Lake Ontario. The majority of the lands of 'upper' Swansea are 30 to 40 feet higher than this, with steep hillsides along Grenadier Pond, Humber River and Rennie Pond.