Local LLMs for Knowledge Management: A RAG Experiment

*This blog was not generated by AI.

While busy building a new Knowledge Management System based on my research paper and obviously using the most popular AI LLMs out there to assist with technical implementation, I decided to experiment a bit.

I completed two courses first:

• Generative AI for Beginners. Instructor: Aakriti E-Learning Academy
• Prompt Engineering for Everyone Bootcamp. Instructor: Navid Shirzadi, PhD, P.Eng.

I wanted to achieve two things:

• What is this Prompt Engineering all about?
• Is it possible to create your own local LLM environment for RAG?

Well, after completing these courses, it opened up a new world. Regardless of how you define or understand AI, it is here to stay and serves as a powerful tool to support and enhance critical thinking. Note: assist, not dictate. Do not let AI use you; use AI to assist you (Human in the loop).

After completing the two courses to set the foundation for what I want to achieve, I used the knowledge gained to make it my own by changing quite a bit of the functionality and technology used. It was a good starting point.

I’m not the first person to do this, but I found it quite interesting.

After using my own local LLM (Opensource LLM), I asked it a question about the ITTI model. (I had to tweak the code quite a few times to ensure it only looks at the PDFs I uploaded, thereby only using the LLM as the “engine” to generate content based on the PDF content, no external content.) Here is how it looks when asking: “Tell me about the ITTI model” (A model I created in my research paper).

Spot on! Only one small error, it should be Interpersonal-Information-Technology

“🧠 Answer

According to the provided document, the ITTI (Information Technology Trust Inventory) model is a framework for measuring trust within organizations. It differs from the Technology Acceptance Model (TAM) in that it takes into account socio-psychological trust attributes between humans.

The ITTI model was selected from verified trust measurement models and extended to include Information Technology Trust Capital (ITTC). The common denominator between the ITTI model and trust capital is the ability to gather survey results based on a human's business role within the organization. These results can be measured as Interpersonal-Information-Tech Trust Capital using weighted measures.

The proposed ITTI model has significant theoretical and practical implications, particularly in leveraging active engagement and technology experiences to foster transparency and support broader trust dynamics (p. 183).”

What I found quite interesting—and funny—is that I asked an external LLM this question:

For me, the future will be local/internal small LLMs.

Each small LLM will need to be fine-tuned in such a way that it only creates content based on the user’s intellectual subject matter, for example, Knowledge Management or a Research Assistant. What I gathered from the prompt engineering courses is that the “role” you specify in a prompt is always the starting point for every prompt. (I have hard-coded the role I want to use; therefore, I can only ask questions, and the LLM already knows which role to assume.)

Benefits?

· Your intellectual property remains secure and private

· It helps you gain deeper insights into topics

· It is local, with no cost involved

· It saves you time, as information retrieval is much faster

· It supports new idea generation

· No technology updates, meaning you won’t lose any functionality. The system remains familiar, with no sudden UI changes or unexpected behaviour.

· It has “memory” capabilities, as the uploaded PDFs remain stored as a source in a “database,” so there is no need to upload them again.

Risks:

· And yes, the risk is definitely there that it will create cognitive decline in humans. More and more studies are emerging about this topic.

· You still need to review and verify the information generated; you need to be a subject matter expert to be able to do this.

· The guardrail included in the code is designed to prevent the model from generating its own content and to ensure it only uses the supplied information; however, this does not guarantee that it will not hallucinate.

· This initial prototype is still in its early stages and is intended only as an experiment.

· My laptop hardware is not optimal, but to be honest, I’m quite impressed by how quickly it responds to a prompt.