Contents
In the last four decades, IT computing has evolved dramatically from Mainframe using COBOL to Artificial Intelligence (AI) / Machine Language (ML), using OpenAI. AI will no doubt revolutionize many industries, and it is essential in the field of software development to prepare for what’s ahead.
The evolution of AI
Neural networks transmit data in the form of input values and output values through connections. Deep learning is a subset of Machine Learning (ML), where artificial neural networks algorithms—modeled to work like the human brain—learn from large amounts of data.
Wikipedia defines AI as “intelligence demonstrated by computers, as opposed to human or animal intelligence. The field of AI research was born at a workshop at Dartmouth College in 1956. The attendees became the founders and leaders of AI research.”
My past experience with AI
Looking back, my first experience with AI was during my undergrad studies when I read the writings of John von Neumann in the late 1980s. Now, concepts that were just a dream back then have come true in real life with AI.
AI/ML have been researched aggressively for the last 7+ years in the software industry. But, the real breakthrough came with open source OpenAI, as depicted in the graphic below.
By early 2023, Microsoft made a multi-billion dollar investment in OpenAI as its key supporter. As a result, OpenAI is heavily leveraged in Microsoft products—Windows 11, GitHub Copilot, Visual Studio, etc.
Experimenting with AI
This week, I looked into the Top 50 technology framework/tools used in the Generative AI application landscape.
Based on the recent coding experiences of model infrastructure and code generation, software engineering is a highly promising field for the use of AI.
The benefits of LangChain
OpenAI is a clear winner in the recent explosive growth of AI. However, two limitations were identified in its current design.
- Closed data model
- Local source integration
LangChain resolves these limitations as depicted below in the logical architecture diagram.
Put simply (and not underestimating the power of technology), LangChain acts as the middleware between the industry LLMs (like OpenAI, HuggingFace, etc.) and supported external data sources (like Google drive, Database, Local files, Wikipedia, etc.)
LangChain is an emerging open source framework with exponential growth. In a short period of time, this growth has become evident by looking at GitHub source code metrics.
Testing LangChain
After trying a dozen types of integrations, it was more fun to learn the core strengths of LangChain technology.
The use case below shows the technical workflow of external .pdf document integration with LangChain/OpenAI frameworks.
In this six-step process, Langchain is the mastermind to load the source data, split the content using MapReduce and other tools/integrations. Content is vectorized using Pinecone open source vector database. OpenAI is used for prompt engineering.
It was mesmerizing to get an intelligent result with the combinations of AI deep learning, natural language processing (NLP), etc.
This result is a testimonial to this unstructured and non-straightforward human intelligence query from the given PDF scope.
Technical references
Also, see LangChain integration sample code.
Conclusion
There is no doubt that Artificial Intelligence will be the next wave of computing, similar to Mobile and Cloud computing. Both of them had an initial tough time getting widely adopted in the industry; but now they are used everywhere in the world, including by competing companies.
AI is heading in the same direction!
Share this article:
