The Convergence of ChatGPT and Data Fabric for the Future of Business Data
By Ananth Chakravarthy,
Artificial intelligence (AI) models are the latest technology driving the world’s digital transformation. These models, which span a range of capabilities and applications, are disrupting the way in which we interface with information, access it, and process it. An important subset of these models, large language models (LLMs) have names that are becoming a regular part of our daily lives: ChatGPT, OPT, CodeGen, and PaLM 2.
In the short time since they were first introduced, LLMs have already surpassed expectations. Their ability to collate and interpret vast quantities of information has enabled them to generate wide-ranging natural responses. With a sufficiently detailed set of instructions, they can now prepare presentations, create artwork, and even write usable code. As a result a wide variety of tasks are now being automated, and they are being performed far more quickly and efficiently than if they were performed by people.
When ChatGPT Meets Data Fabric
With these capabilities in mind, we must look at the role of unified data access interfaces in expanding this functionality. Data fabric is an architectural approach that enables businesses to gather information cohesively, no matter where the information is sourced from. All too often, it will be derived from a disparate set of data caches and repositories – on-premises databanks, data lakes, off-site silos and storage facilities, and in the cloud through SaaS applications like Salesforce or Workday. Data fabric establishes a single point of data access while ensuring that the organisation’s data governance and security regulations are followed.
With the power of ChatGPT, data fabric can enable users to access information by expressing their needs in natural language. Although this technology is still at a nascent stage of development and is only available through early-access programmes, it has already shown great potential.
This is how it works: The user describes what he or she needs using natural language within predefined data catalogues. A ChatGPT API then automatically generates a SQL query that is immediately sent to the data fabric query engine. ChatGPT concludes the process by explaining, in simple terms, how the query was generated, enabling validation. This process only requires accessing a single system, as the data fabric provides secure, traceable access to all data repositories.
Enabling Seamless Data Access
As technology continues to advance and new breakthroughs are made, these interactions are only poised to become more common, and platforms that integrate data fabric with ChatGPT and other similar models are at the forefront of this technological revolution. AI and machine learning (ML) capabilities have been an integral part of data fabric for years, with ML models being used to analyse incoming questions from users and rapidly suggesting relevant, context-specific responses. This is similar to Amazon’s customised list of suggestions for every user profile, or friends suggestions on social media platforms.
ML models are also utilized by data fabric to analyse other queries made by customers in the past, which helps in the formulation of responses to future queries, ultimately driving the automation of key data management tasks. As data fabric architecture advances, obtaining accurate answers becomes easier and faster. In order to expand the use of LLMs and tackle larger workloads, organisations are starting to explore new techniques, such as automatically proposing ways to combine data from diverse sources when building data products in data engineering tasks, or recommending suitable business-friendly names for artefacts. Another potential application could be automatically incorporating semantic tags in the data catalogue, such as tagging items containing personally identifiable information.
Leading analysts in the field of data science have also recognised the potential of data fabric. In the ebook entitled “Understand the Role of Data Fabric — Guides for Effective Business Decision Making,” Gartner predicts that by 2024 data fabric deployments will quadruple efficiency in data utilization while halving human-driven data management tasks.
In a business context, data fabric gives data scientists integrated access to curated, governed data. Such data serves as the foundation for generating ML models that cater to a variety of purposes, ranging from automatically developing prices and personalized offers for individual customers to analysing churn rates and segmenting customers based on their profiles.
In the years to come, we will probably be seeing AI and ML technologies playing an even more critical role in business operations, driven by the convergence of ChatGPT and data fabric.
(The author is Ananth Chakravarthy, RVP Sales, Denodo India, and the views expressed in this article are his own)
