Christian Gralingen
Generative AI presents an organizational puzzle. Businesses have collectively invested billions of dollars to give employees access to general-purpose large language models (LLMs) to enhance personal productivity while in most cases struggling to develop and adopt more strategic applications of the technology. Meaningful return on investment, much less competitive advantage, is likely to remain elusive unless companies can use GenAI to make innovative process improvements that scale across functions and business units.1
Our interviews with 87 practitioners in 23 large organizations revealed that leaders who scale value creation with generative AI cultivate three key practices. First, they expand the scope of use cases across processes rather than remaining focused on a specific task. Second, they treat each use case as a work in progress to be continually improved. And third, they quickly identify and abandon use cases that fail to bring measurable value to the organization.
However, most traditional companies are not structured to institutionalize these three practices. Many operate as multidivisional organizations characterized by multiple profit and loss units with duplicated functions, limited cross-functional information flow, and internal competition for resources.2 This setup makes it difficult to scale generative AI use cases across processes and units.
In our research, we found that the few leaders who are overcoming these challenges are moving beyond the classical hub-and-spoke models that many organizations have used to connect centralized AI technical expertise to each unit. They are developing a new kind of internal resource that we call the AI spine. It provides a flexible core structure for implementing, evolving, and abandoning LLM use cases at scale, keeping the generative AI portfolio both focused and current. Notably, rather than deploying technologists out into business units, as is commonly done, this structure pulls individuals with domain knowledge of business processes into the core and makes them part of the team.
A retail bank that we studied demonstrates the kind of scaling and value creation that an AI spine supports. Initially, the bank’s AI spine spearheaded the implementation of an email assistant for customer service employees. In an early, limited rollout, those using the assistant collectively saved about 700 hours. Once the tool was put into wider use, it reduced email handling time by 15%, allowing employees to dedicate more time to managing complex cases. Encouraged by that success, the spine oversaw the implementation of LLM-powered email thread summaries, call transcriptions, and analyses. That yielded data that provided new insights for customer service employees, leading the bank to start reengineering its approach to customer relationship management. The data was also used to develop the next iterations of the email assistant.
We observed another example at a medical coding company that applies standardized alphanumeric codes to documentation related to diagnoses, treatments, and procedures for the purposes of insurance and health care management. There, the AI spine provided a structure that allowed it to turn its first LLM application for automated coding into a new line of business. An internal LLM application had reduced coding time from 25 minutes per case to 2 seconds, cutting the cost of coding by 60% compared with having humans doing the work. (Affected staff members were able to take on other responsibilities.) The AI spine was able to build out the application into a product for insurance companies that need to verify whether bills have been correctly coded, increasing the company’s reach into the medical insurance market and creating a new revenue stream.
Connecting AI Efforts Across the Enterprise
The AI spine is a cross-functional backbone that is dedicated to diffusing and scaling LLM use cases across the organization, focusing on reducing duplicative efforts and achieving economies of scale as solutions expand across business units and processes. Because it is a central point for collaboration between technologists and those holding business domain knowledge, the spine holds the expertise required for rewiring and continually improving processes end to end and across functions, as in the bank example above. As we found at the medical coding company, this structure can be implemented not only in large organizations but also in small and medium-sized enterprises.
In the cases we observed, funding for the AI spine was allocated by top management, and the spine also got a cut of increased revenue or costs savings resulting from applications deployed. That mechanism creates the right incentives: It forces the organization to measure ROI, stay focused, and avoid disproportionate spending on “convenience” use cases that seem useful but don’t materially affect costs or revenue. By being independently funded, the spine maintains decision-making autonomy vis-à-vis other divisions so it can identify and encourage use cases that have the potential to improve processes cutting across divisions. A more traditional AI center of excellence with a hub-and-spoke structure is more likely to focus on cases within individual business units rather than across them.
The spine is overseen by a C-suite leader, who keeps its efforts aligned with overall strategic objectives; this may be a chief technology officer or chief digital officer, or one of their direct subordinates. Sitting within the structure are AI developers, risk and compliance personnel, and a technology owner. (See “The AI Spine.”)
The technology owner is responsible for preventing the fragmentation of data flows and tools. They typically oversee the creation of centralized data platforms, prompt libraries, models, and evaluation and technical performance metrics (including token consumption and financial costs). This oversight reduces rework and lowers marginal costs as applications are diffused across the organization.
