Navigating AI-Driven Semantic Text Analysis for Decision Making in Idea Management

_{Created with DALL-E3}

In modern organisations, the identification and use of innovative ideas for process improvements or new products from the workforce, for example, is of crucial importance, but the implementation of such approaches requires a great deal of effort [1]. So-called idea management systems are used here, through which employees submit their ideas primarily in text form, which are then analysed and evaluated in order to make decisions on which to continue or reject [2]. However, organisations can face considerable challenges in idea management when it comes to ensuring an appropriately detailed analysis and processing of the ideas submitted [3]. With a high level of employee participation, a large number of ideas can be submitted, which often leads to similar or almost identical ideas being submitted multiple times. This results in an increased evaluation effort and complicates the decision-making process regarding the further pursuit of ideas. Experts or managers with a high level of technological and specialist knowledge are also used as part of the evaluation processes. This is a group of people whose cost intensity requires a high financial outlay and whose time availability is generally very limited.

This is where the Pathfinder project comes in by investigating the current possibilities in the field of (generative) artificial intelligence (AI) for semantically analysing texts using the example of idea management systems. The aim is to evaluate the use of AI to support and (partially) automate processes for analysing, evaluating and prioritising ideas. Modern AI technologies should help to recognise semantic duplicates and compare similar ideas with each other [4]. This makes it possible to reduce redundant proposals and network idea providers with similar approaches, which in turn supports collaborative development and the exploitation of synergy potential and can lead, for example, to the establishment of so-called communities of practice [5]. The research question of the extent to which AI, in particular generative AI and language models, can be used to extract semantic information from text-based content is at the centre of the investigations. The potential of these technologies to support the complex and resource-intensive process of idea evaluation is high [6], but the Pathfinder project is paying particular attention to ensuring that the quality of the analyses and evaluations is not compromised and remains the same or is improved in collaboration with humans.

To begin with, we conduct a systematic literature review based on the method of Xiao and Watson [7]. The aim is to determine the current status of AI-supported semantic text analysis, to analyse the possible applications and the state of research of AI in the field of idea management and to explore relevant metrics for evaluating such solutions. We then intend to test and evaluate the most promising approaches with a prototype idea management system. For this purpose, we need as much data as possible. We plan to use the following two data sources:

1. The Digital and Data Literacy in Teaching Lab (DDLitLab), which provides funding for this student project, has received numerous project applications for funding over the past three years. These applications are used as a data source to find commonalities between them, for example through thematic focus or methodological approaches. This allows matchmaking to take place, potentially resulting in joint funding or collaborations.
2. The prototype idea management system will be used to collect ideas for improving sustainability at Universität Hamburg during two planned workshops. In addition, a generative AI creates synthetic ideas that serve as a control instance. These real and synthetic ideas are then compared with each other to assess their potential, with both humans and an AI categorising the ideas independently of each other. The aim is to analyse the differences or similarities in the assessments of humans and AI.

Data plays an essential role in our research project. In order to support further research in this area in the future and to offer other research groups the opportunity to evaluate their own algorithms, we intend to create a dataset and make it available to the research community as an open source resource. This dataset will consist of the ideas generated and collected as part of the project, but excluding the project applications mentioned under point 1 for data protection reasons. In this way, we will create a sound data basis that will make it possible to track our research results and compare them with new approaches.

Another key aspect of our research is ensuring fairness in evaluation and decision-making. When using artificial intelligence, there is always a risk that the algorithms will lead to unbalanced or unfair results due to bias in the training data or the models used [8], [9]. This is particularly relevant in the automated evaluation of ideas, as bias could potentially penalise or favour innovative approaches. Our aim is to explore methods and approaches that prevent or at least highlight such biases. This should help to ensure that AI-based processing remains as objective and transparent as possible and that ideas are evaluated fairly.

The Pathfinder project thus aims to contribute to the further development of idea management systems through the targeted use of modern AI technologies. Our approach aims to increase the efficiency and quality of idea evaluation, promote collaboration between idea providers and at the same time encourage fairness and transparency. In this way, we create a basis for organisations to better exploit the dormant creative potential of their employees and establish innovation processes.

And here is a short video presentation of the project:

Literature:

[1] S. Høyrup, „Employee-driven innovation and workplace learning: basic concepts, approaches and themes“, Transfer: European Review of Labour and Research, Bd. 16, Nr. 2, S. 143–154, 2010, doi: 10.1177/1024258910364102. 
[2] C. Sandstrom und J. Bjork, „Idea management systems for a changing innovation landscape“, IJPD, Bd. 11, Nr. 3/4, S. 310, 2010, doi: 10.1504/IJPD.2010.033964. 
[3] J. Just, T. Ströhle, J. Füller, und K. Hutter, „AI-based novelty detection in crowdsourced idea spaces“, Innovation, Bd. 26, Nr. 3, S. 359–386, 2024, doi: 10.1080/14479338.2023.2215740. 
[4] S. Leka, „The Role of Artificial Intelligence in Idea Management Systems and Innovation Processes: An Integrative Review“, in AICCONF ’24: Proceedings of the Cognitive Models and Artificial Intelligence Conference, 2024. doi: https://dl.acm.org/doi/10.1145/3660853.3660890. 
[5] E. Davenport und H. Hall, „Organizational knowledge and communities of practice“, Annual review of Information Science and technology, Bd. 36, Nr. 1, 2002, doi: 10.1002/aris.1440360105. 
[6] J. Bell, C. Pescher, G. Tellis, und J. Füller, „Can AI Help in Ideation? A Theory-Based Model for Idea Screening in Crowdsourcing Contests“, 2023, doi: 10.1287/mksc.2023.1434. 
[7] Y. Xiao und M. Watson, „Guidance on Conducting a Systematic Literature Review“, Journal of Planning Education and Research, Bd. 39, Nr. 1, S. 93–112, 2019, doi: 10.1177/0739456X17723971. 
[8] N. Mehrabi, F. Morstatter, N. Saxena, K. Lerman, und A. Galstyan, „A Survey on Bias and Fairness in Machine Learning“, 2022, Verfügbar unter: http://arxiv.org/abs/1908.09635 
[9] P. S. Varsha, „How can we manage biases in artificial intelligence systems – A systematic literature review“, International Journal of Information Management Data Insights, Bd. 3, Nr. 1, S. 100165, 2023, doi: 10.1016/j.jjimei.2023.100165.