To say that AI is complicated is an understatement. Machine learning, a subset of artificial intelligence, is a multifaceted process that integrates and scales mountains of data that comes in different forms from various sources. Data is used to train machine learning models in order to develop insights and solutions from newly acquired related data. For example, an image recognition model trained with several million dog and cat photos can efficiently classify a new image as either a cat or a dog.
A better way to build and manage machine learning models
The development of machine learning models requires the coordination of many processes linked together with pipelines. Pipelines can handle data ingestion, scrubbing, and manipulation from varied sources for training and inference. Machine learning models use end-to-end pipelines to manage input and output data collection and processing.
To deal with the extraordinary growth of AI and its ever-increasing complexity, IBM created an open-source framework called CodeFlare to deal with AI’s complex pipeline requirements. CodeFlare simplifies the integration, scaling, and acceleration of complex multi-step analytics and machine learning pipelines on the cloud. Hybrid cloud deployment is one of the critical design points for CodeFlare, which using OpenShift can be easily deployed from on-premises to public clouds to edge.
It is important to note that CodeFlare is not currently a generally available product, and IBM has yet to commit to a timeline for it becoming a product. Nevertheless, CodeFlare is available as an open-source project. And, as an evolving project, some aspects of orchestration and automation are still work in progress. At this stage, issues can be reported through the public GitHub project. IBM invites community engagement through issue and bug reports, which will be handled on a best effort basis.
CodeFlare’s main features are:
- Pipeline execution and scaling: CodeFlare Pipelines facilitate the definition and parallel execution of pipelines. It unifies pipeline workflows across multiple frameworks while providing nearly optimal scale-out parallelism on pipelined computations.
- Deploy and integrate anywhere: CodeFlare simplifies deployment and integration by enabling a serverless user experience with Red Hat OpenShift and IBM Cloud Code Engine and providing adapters and connectors to make it simple to load data and connect to data services.
CodeFlare is built on top of Ray, an open-source distributed computing framework for machine learning applications. According to IBM, CodeFlare extends the capabilities of Ray by adding specific elements to make scaling workflows easier. CodeFlare pipelines run on a serverless platform using IBM Cloud Code Engine and Red Hat OpenShift. This platform provides CodeFlare the flexibility to be deployed just about anywhere.
CodeFlare can integrate emerging workflows with complex pipelines that require integration and coordination of different tools and runtimes. It is designed also to scale complex pipelines such as multi-step NLP, complex time series and forecasting, reinforcement learning, and AI-Workbenches. The framework can integrate, run, and scale heterogenous pipelines that use data from multiple sources and require different treatments.
How much difference does CodeFlare make?
According to the IBM Research blog, CodeFlare significantly increases the efficiency of machine learning. The blog states that a user used the framework to analyze and optimize approximately 100,000 pipelines for training machine learning models. CodeFlare cut the time it took to execute each pipeline from 4 hours to 15 minutes - an 18x speedup provided by CodeFlare.
The research blog also indicates that CodeFlare can save scientists months of work on large pipelines, providing the data team more time for productive and development work.
Studies show that about 75% of prototype machine learning models fail to transition to production status despite large investments in artificial intelligence. Several reasons for low conversion rates range from poor project planning to weak collaboration and communications between AI data team members.
CodeFlare is a purpose-built platform that provides complete end-to-end pipeline visibility and analytics for a broad range of machine learning models and workflows. It provides a more straightforward way to integrate and scale full pipelines while offering a unified runtime and programming interface.
For those reasons, despite the historical high AI model failure rates, Moor Insights & Strategy believes that machine learning models using CodeFlare pipelines will have a high percentage of machine learning models transition from experimental status to production status.
- IBM expects to enhance CodeFlare to support increasingly more complex pipelines.
- Future development plans are expected to include enhanced fault-tolerance and support for pipeline visualization.
- IBM has made CodeFlare available at the GitHub project CodeFlare repository. There are also examples that run on IBM Cloud and Red Hat OpenShift.
Note: Moor Insights & Strategy writers and editors may have contributed to this article.