Experimenting with LangChain AI

Ganesan Senthilvel
Ganesan SenthilvelSenior Engineering Manager

In the last four decades, IT computing has evolved dramatically from Mainframe using COBOL to Artificial Intelligence (AI) / Machine Language (ML), using OpenAI. AI will no doubt revolutionize many industries, and it is essential in the field of software development to prepare for what’s ahead.

Computing Timeline
The evolution of computing.

The evolution of AI

Neural networks transmit data in the form of input values and output values through connections. Deep learning is a subset of Machine Learning (ML), where artificial neural networks algorithms—modeled to work like the human brain—learn from large amounts of data.

Computing Pyramid
The evolution of AI.

Wikipedia defines AI as “intelligence demonstrated by computers, as opposed to human or animal intelligence. The field of AI research was born at a workshop at Dartmouth College in 1956. The attendees became the founders and leaders of AI research.”

My past experience with AI

Looking back, my first experience with AI was during my undergrad studies when I read the writings of John von Neumann in the late 1980s. Now, concepts that were just a dream back then have come true in real life with AI.

AI/ML have been researched aggressively for the last 7+ years in the software industry. But, the real breakthrough came with open source OpenAI, as depicted in the graphic below.

Open Source AI
Open source AI timeline.

By early 2023, Microsoft made a multi-billion dollar investment in OpenAI as its key supporter. As a result, OpenAI is heavily leveraged in Microsoft products—Windows 11, GitHub Copilot, Visual Studio, etc.

ChatGPT Interface
A ChatGPT user interface.

Experimenting with AI

This week, I looked into the Top 50 technology framework/tools used in the Generative AI application landscape.

Top 50 AI frameworks
Source: Tulsi Soni @shedntcare_

Based on the recent coding experiences of model infrastructure and code generation, software engineering is a highly promising field for the use of AI.

The benefits of LangChain

OpenAI is a clear winner in the recent explosive growth of AI. However, two limitations were identified in its current design.

  1. Closed data model
Closed Data Model
  1. Local source integration
Local Source Integration

LangChain resolves these limitations as depicted below in the logical architecture diagram.

LangChainn Architecture

Put simply (and not underestimating the power of technology), LangChain acts as the middleware between the industry LLMs (like OpenAI, HuggingFace, etc.) and supported external data sources (like Google drive, Database, Local files, Wikipedia, etc.)

LangChain is an emerging open source framework with exponential growth. In a short period of time, this growth has become evident by looking at GitHub source code metrics.

LangChain Metrics
GitHub metrics for LangChain

Testing LangChain

After trying a dozen types of integrations, it was more fun to learn the core strengths of LangChain technology.

The use case below shows the technical workflow of external .pdf document integration with LangChain/OpenAI frameworks.

External PDF Integration
Workflow for integrating a .pdf

In this six-step process, Langchain is the mastermind to load the source data, split the content using MapReduce and other tools/integrations. Content is vectorized using Pinecone open source vector database. OpenAI is used for prompt engineering.

It was mesmerizing to get an intelligent result with the combinations of AI deep learning, natural language processing (NLP), etc.

PDF Integration Results
AI answering questions based on input from a .pdf file

This result is a testimonial to this unstructured and non-straightforward human intelligence query from the given PDF scope.

Technical references

LangChain Technical Reference

Also, see LangChain integration sample code.


There is no doubt that Artificial Intelligence will be the next wave of computing, similar to Mobile and Cloud computing. Both of them had an initial tough time getting widely adopted in the industry; but now they are used everywhere in the world, including by competing companies.

AI is heading in the same direction!

Share this article:

You May Also Like

New Frameworks Provide New Ways to Create Cross Platform, Cross Mobile Applications

| By Ganesan Senthilvel

My recent experiences working with Flutter and Capacitor.

How to Help Guide Developers Through Your API Documentation

| By Chris Johnson

Tips on how to go beyond just documenting API endpoints and parameters.