Semantic Text Transmission via Prediction with Small Language Models: Cost-Similarity Trade-off
2024 (English)In: 2024 IEEE INTERNATIONAL CONFERENCE ON COMMUNICATIONS WORKSHOPS, ICC WORKSHOPS 2024, IEEE , 2024, p. 1244-1249Conference paper, Published paper (Refereed)
Abstract [en]
We consider the communication of natural language text from a source to a destination over noiseless and character-erasure channels. We exploit language's inherent correlations and predictability to constrain transmission costs by allowing the destination to predict or complete words with potential dissimilarity with the source text. Concretely, our objective is to obtain achievable ((c) over bar, (s) over bar) pairs, where (C) over bar is the average transmission cost at the source and (s) over bar is the average semantic similarity measured via cosine similarity between vector embedding of words at the source and those predicted/completed at the destination. We obtain ((c) over bar, (s) over bar) pairs for neural language and first-order Markov chain-based small language models (SLM) for prediction, using both a threshold policy that transmits a word if its cosine similarity with that predicted/completed at the destination is below a threshold, and a periodic policy, which transmits words after a specific interval and predicts/completes the words in between, at the destination. We adopt an SLM for word completion. We demonstrate that, when communication occurs over a noiseless channel, the threshold policy achieves a higher (s) over bar for a given (c) over bar than the periodic policy and that the (s) over bar achieved with the neural SLM is greater than or equal to that of the Markov chain-based algorithm for the same (c) over bar. The improved performance comes with a higher complexity in terms of time and computing requirements. However, when communication occurs over a character-erasure channel, all prediction algorithms and scheduling policies perform poorly. Furthermore, if character-level Huffman coding is used, the required (c) over bar to achieve a given <overline>s is reduced, but the above observations still apply.
Place, publisher, year, edition, pages
IEEE , 2024. p. 1244-1249
Series
IEEE International Conference on Communications Workshops, ISSN 2164-7038, E-ISSN 2694-2941
National Category
Telecommunications
Identifiers
URN: urn:nbn:se:liu:diva-210352DOI: 10.1109/ICCWORKSHOPS59551.2024.10615608ISI: 001296276700207ISBN: 9798350304060 (print)ISBN: 9798350304053 (electronic)OAI: oai:DiVA.org:liu-210352DiVA, id: diva2:1920017
Conference
59th Annual IEEE International Conference on Communications (IEEE ICC), Denver, CO, jun 09-13, 2024
Note
Funding Agencies|Swedish Research Council (VR), Excellence Center at Linkoping - Lund in Information Technology (ELLIIT); European Union [101096526]; European Union's Horizon Europe research and innovation programme under the Marie Sklodowska-Curie Grant [101131481]
2024-12-102024-12-102024-12-10