liu.seSearch for publications in DiVA
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • oxford
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Digital Twin-Enabled Adaptive Robotics: Leveraging Large Language Models in Isaac Sim for Unstructured Environments
Linköping University, Department of Management and Engineering, Product Realisation. Linköping University, Faculty of Science & Engineering. (Design Automation Laboratory)ORCID iD: 0000-0003-1745-3869
Linköping University, Department of Management and Engineering, Product Realisation. Linköping University, Faculty of Science & Engineering. (Design Automation Laboratory)
Linköping University, Department of Management and Engineering, Product Realisation. Linköping University, Faculty of Science & Engineering. (Design Automation Laboratory)ORCID iD: 0009-0000-4905-2344
Linköping University, Department of Management and Engineering, Product Realisation. Linköping University, Faculty of Science & Engineering. (Design Automation Laboratory)
Show others and affiliations
2025 (English)In: Machines, E-ISSN 2075-1702, Vol. 13, no 7, article id 620Article in journal (Refereed) Published
Abstract [en]

As industrial automation evolves towards human-centric, adaptable solutions, collaborative robots must overcome challenges in unstructured, dynamic environments. This paper extends our previous work on developing a digital shadow for industrial robots by introducing a comprehensive framework that bridges the gap between physical systems and their virtual counterparts. The proposed framework advances toward a fully functional digital twin by integrating real-time perception and intuitive human–robot interaction capabilities. The framework is applied to a hospital test lab scenario, where a YuMi robot automates the sorting of microscope slides. The system incorporates a RealSense D435i depth camera for environment perception, Isaac Sim for virtual environment synchronization, and a locally hosted large language model (Mistral 7B) for interpreting user voice commands. These components work together to achieve bi-directional synchronization between the physical and digital environments. The framework was evaluated through 20 test runs under varying conditions. A validation study measured the performance of the perception module, simulation, and language interface, with a 60% overall success rate. Additionally, synchronization accuracy between the simulated and physical robot joint movements reached 98.11%, demonstrating strong alignment between the digital and physical systems. By combining local LLM processing, real-time vision, and robot simulation, the approach enables untrained users to interact with collaborative robots in dynamic settings. The results highlight its potential for improving flexibility and usability in industrial automation.

Place, publisher, year, edition, pages
2025. Vol. 13, no 7, article id 620
Keywords [en]
Adaptive digital twin; Human-robot collaboration (HRC); Adaptive robotics
National Category
Production Engineering, Human Work Science and Ergonomics
Identifiers
URN: urn:nbn:se:liu:diva-216601DOI: 10.3390/machines13070620ISI: 001535521300001Scopus ID: 2-s2.0-105011624230OAI: oai:DiVA.org:liu-216601DiVA, id: diva2:1989566
Funder
Vinnova, 2021-02481Vinnova, 2023-02674Available from: 2025-08-18 Created: 2025-08-18 Last updated: 2026-02-10
In thesis
1. Adaptive Automation Strategies for Increasing Variability in Design and Production
Open this publication in new window or tab >>Adaptive Automation Strategies for Increasing Variability in Design and Production
2026 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

The increasing demand for customized products, together with the need for flexible, human-centric, and resilient manufacturing systems has intensified the need for automation solutions capable of operating in dynamic and unstructured industrial environments. This dissertation shows how automation methodologies evolve and where they fail as complexity and variability increase across the design and production domains.

The research begins by addressing time-consuming and iterative engineering tasks through design automation. Using Knowledge-Based Engineering (KBE) approaches, automated frameworks were developed to streamline engineering workflows and support consistent decision-making in structured industrial settings. However, when extending the focus to real-world production, the growing complexity and uncertainty of the environment create substantial challenges for both design and production automation.

While KBE provides structure and consistency, its reliance on predefined rules necessitates standardization, resulting in a rigid design space and limited adaptability. To overcome these inherent restrictions, the research integrates complementary tools and techniques that enable more flexible and adaptive automation. Camera vision captures real-world conditions and tracks changes in the environment, while large language models, combined with an agent-based approach, provide reasoning capabilities that interpret variations in products or processes and generate adaptive decision-making strategies. Digital twin simulations validate and predict the outcomes of these variations in a virtual environment, allowing the system to respond proactively and safely by reconciling real-time data with simulation outcomes.

Overall, this work contributes a holistic and scalable automation methodology that unifies design automation, adaptive digital twins, and knowledge-driven reasoning. The results demonstrate how structured engineering knowledge, combined with reasoning and adaptive technologies, enables the development of resilient automation solutions for the increasingly unstructured landscape of future Industry.

Abstract [sv]

Den ökande efterfrågan på kundanpassade produkter, tillsammans med behovet av flexibla, människocentrerade och motståndskraftiga tillverkningssystem, har intensifierat behovet av automatiseringslösningar som kan fungera i dynamiska och ostrukturerade industriella miljöer. Denna avhandling visar hur automatiserings-metoder utvecklas och var de misslyckas i takt med att komplexitet och variation ökar inom design- och produktionsområdena.

Forskningen börjar med att hantera tidskrävande och iterativa tekniska uppgifter genom designautomation. Med hjälp av kunskapsbaserad teknik (KBE) utvecklades automatiserade ramverk för att effektivisera tekniska arbetsflöden och stödja konsekvent beslutsfattande i strukturerade industriella miljöer. Men när fokus utvidgas till verklig produktion skapar den växande komplexiteten och osäkerheten i miljön betydande utmaningar för både design- och produktionsautomation.

Medan KBE ger struktur och konsekvens, kräver dess beroende av fördefinierade regler standardisering, vilket resulterar i ett rigid designutrymme och begränsad anpassningsförmåga. För att övervinna dessa inneboende begränsningar integrerar forskningen kompletterande verktyg och tekniker som möjliggör mer flexibel och adaptiv automatisering. Kameraseende fångar verkliga förhållanden och spårar förändringar i miljön, medan stora språkmodeller, i kombination med en agentbaserad metod, ger resonemangsförmåga som tolkar variationer i produkter eller processer och genererar adaptiva beslutsstrategier. Digitala tvillingsimuleringar validerar och förutsäger resultaten av dessa variationer i en virtuell miljö, vilket gör att systemet kan reagera proaktivt och säkert genom att förena realtidsdata med simuleringsresultat.

Sammantaget bidrar detta arbete med en holistisk och skalbar automationsmetodik som förenar designautomation, adaptiva digitala tvillingar och kunskapsdrivet resonemang. Resultaten visar hur strukturerad ingenjörskunskap, i kombination med resonemang och adaptiva teknologier, möjliggör utveckling av motståndskraftiga automationslösningar för det alltmer ostrukturerade landskapet inom framtidens industri.

Place, publisher, year, edition, pages
Linköping: Linköping University Electronic Press, 2026. p. 62
Series
Linköping Studies in Science and Technology. Dissertations, ISSN 0345-7524 ; 2505
National Category
Production Engineering, Human Work Science and Ergonomics
Identifiers
urn:nbn:se:liu:diva-221174 (URN)10.3384/9789181184624 (DOI)9789181184617 (ISBN)9789181184624 (ISBN)
Public defence
2026-03-06, ACAS, A-building, Campus Valla, Linköping, 09:15 (English)
Opponent
Supervisors
Available from: 2026-02-10 Created: 2026-02-10 Last updated: 2026-02-11Bibliographically approved

Open Access in DiVA

fulltext(11043 kB)17 downloads
File information
File name FULLTEXT02.pdfFile size 11043 kBChecksum SHA-512
109f4eaa3ca08736850979e081ad1aeb89cf87950e8ee609243ee86db5b027b11871d1f355cea9f5e4dbcf5744d722e6eb1c8436bdb7e87e1233a08addae1b9e
Type fulltextMimetype application/pdf

Other links

Publisher's full textScopus

Authority records

Ikechukwu, Oscar Chigozie

Search in DiVA

By author/editor
Nambiar, SanjayPaul, Rahul ChiramelIkechukwu, Oscar ChigozieJonsson, MarieTarkian, Mehdi
By organisation
Product RealisationFaculty of Science & Engineering
In the same journal
Machines
Production Engineering, Human Work Science and Ergonomics

Search outside of DiVA

GoogleGoogle Scholar
Total: 17 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

doi
urn-nbn

Altmetric score

doi
urn-nbn
Total: 112 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • oxford
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf