In Sweden, older persons living with severe mental illnesses (SMI) increasingly live independently in the community with support from informal and formal carers and welfare services. A proportion of this group is older people with SMI who, due to age or disability, receive municipal eldercare services such as home care in their ordinary housing, or in residential care. The situation and needs of this group demand that eldercare staff have an understanding for the older person’s situation and the tools to provide appropriate care. The aim of this present study is to explore the experiences of eldercare staff working with older people with SMI. Focus group interviews were conducted with staff from three residential care facilities and two home care teams in one mid-sized Swedish municipality. To analyse the interviews, qualitative content analysis was used. The overarching theme from the analysis was “Doing the best they can,” which was unpacked in two categories: “Working with fragmentary knowledge” and “Finding the right approach.” We could also see differences between the experiences of staff in residential care and in home care. The eldercare staff in our interviews faced lack of formal training, insufficient information and skills concerning mental illness and its treatment. Nevertheless, they tried to make the situation of the older people with SMI as good as possible using experiential and tacit knowledge. Our results also point to system level barriers that hinder effective care for older people with SMI.