Flowdapt is an Emergent Methods project for large-scale adaptive modeling challenges. It is highly adaptable to handle customized workflows (plug-ins) in parallelized environments. And the best part, is that Flowdapt is fully open-source
Flowdapt is for large-scale cluster orchestration, particularly well suited for real-time adaptive modeling. The design principles of Flowdapt include:
🚲 Highly parallelized compute efficiency
🤖 Automatic resource management and sharing
🐞 Rapid (local) prototyping and debuggability
🔌 Intuitive cluster-wide data sharing methods
⏱ Easy scheduling for real-time applications
📝 Intuitive configuration and live configurability
🚚 Deployment cycle efficiency
🔬 Micro-service-first design
🕸 Kubernetes-style schema and behavior
🚀 Vanilla Python, swap effortlessly between Ray, Dask, or Local executors
Flowdapt is built to adopt any application that may require high-level orchestration of compute-heavy libraries (i.e. PyTorch) as well as cluster microservices. Some of the most notable plug-ins include:
AskNews: a contextualized, on-premise, user-specific LLM deployment based on Llama2. AskNews summarizes and indexes all the latest news so that you can ask natural language questions and get and up-to-date distillation of information with hyperlinks and article titles to all the related news sources.
Nowcast: a weather forecasting plug-in designed to provide short-term hyper-local forecasts for hundreds of cities simultaneously. The plug-in collects new meteorological data as it becomes avaialable and retrains models if their performance "drifts."
• CryptoCast: a cryptocurrency market forecast plug-in designed to provide a highly modifiable interface for per-asset model building/updating.
AskArxiv: a contextualized, on-premise LLM deployment that summarizes and indexes all the latest ArXiv articles. Users can ask natural language questions about the latest academic breakthroughs.
Generalizing adaptive-modeling of chaotic market forecasts
Real-time adaptive modeling for finance
FreqAI is an open-source project that allows for time-series forecasting via established machine learning algorithms, such as Reinforcement Learning, Decision Trees, and Neural Networks.
Built in Python, the core engine allows users to connect any open-source machine learning or data analysis library to create a custom system for adaptive modeling. We currently provide 18 pre-configured prediction models for XGBoost, LightGBM, CatBoost, TensorFlow, PyTorch, and Stable Baselines.
DISCLAIMER FreqAI has gained such a strong reputation in the open-source machine learning community that it has become a target of imposter projects aimed at using the FreqAI name to offer cryptocurrencies. FreqAI is not affiliated with any cryptocurrency offerings. FreqAI is, and always will be, a not-for-profit, open-source project. FreqAI does not have a crypto token, FreqAI does not sell signals, and FreqAI does not have a domain besides the freqtrade documentation https://www.freqtrade.io/en/latest/freqai/. Please beware of imposter projects, and help us by reporting them to the official FreqAI discord server.
100% Open Source
Thanks to the open-source community, FreqAI is a robust and well documented software that has been vetted by thousands of users and has more than 15 unique developer contributors.
The community of active users help report bugs and contribute to the FreqAI discord server being an interactive knowledge base with 10k+ posts.
The core engine is, and will always remain, open-source.
Innovative Algorithmic Techniques
The FreqAI core engine facilitates agile development of custom machine learning workflows.
We provide industry standard outlier detection methods together with novel data metrics, and allow for use of any available data manipulation tool whilst ensuring statistically safe data handling.
Adding coherence to the SKLearn pipeline
A Python library for complex data pipelines
DataSieve is an Emergent Methods tool designed to add key features to the SKLearn pipeline, including:
• A variety of custom transforms (Dissimilarity Index, SVM outlier extractor, etc.)
• Outlier removal across your X, y, and sample weights arrays according to simple or complex criteria.
• Feature column removal based on arbitrary criteria (e.g., low variance features.)
• Feature column name changes for certain transformations (e.g., PCA.)
• Outlier classification without removal.
• Backend customization for parallelization (e.g., Dask, Ray, loky, etc.)
DataSieve is deployed in a variety of adaptive-modeling softwares such as FreqAI and Flowdapt.
Modern configuration handling
Reducing friction between configuration and code
Manifest is an Emergent Methods tool designed to improve the flexibility of the python software configuration process. Manifest allows you to:
• Define custom models to validate your configurations.
• Load configurations from a variety of sources, such as environment variables, JSON files, YAML files, or TOML files.
• Instantiate Python objects directly from your configurations.
• Leverage dynamic configurations through expressions.
• Easily serialize and deserialize Manifests based on file extensions.
Manifest is deployed in Flowdapt.
Abstracting low-level LLM processes
An LLM abstraction library providing high-level objects for complex thought strategies
DeepThought is an Emergent Methods library designed to help us easily manipulate high-level LLM abstractions. The library is in active development and it includes:
• Industry standard Tree of Thought, Chain of Thought, and Meta Prompting strategies.
• Experimental thought strategies.
• High-level abstractions on long-term/short-term memory adaptation.
• Adding Flowdapt workflow execution capabilities to LLM thought processes.
Scaling via consensus
A library geared toward efficiently scaling applications that need consensus algorithms.
Adrift is an Emergent Methods tool designed to more efficiently scale duplicate services in need of consensus. The library is in active development and it includes:
JaiRevAI is a style-transfer software based on Convolutional Neural Networks (CNNs).
High-resolution style-transfer is achieved by optimizing the “pixel distance” between a content image and a style image.
JaiRevAI boasts a variety of unique image-handling methods that do not exist anywhere else in the industry. These novel techniques yield stunning ultra-high resolution print-worth imagery that has earned its place hanging in art galleries as well as in numerous homes around the world.
Unique implementation for unique designs
The JaiRevAI software generates unmatched ultra-high-detail textures and colors and allows for stunning and unique style generation.
60+ demonstration pieces, ranging from landscapes to person portraits, are available at emergentartwork.com.
Featured at the M.A.D.S. Art Gallery
Apart from gracing the walls of numerous clients, the art pieces ‘Les Fleurs Perdues’ were displayed at the M.A.D.S. Art Gallery as part of the 2021 ‘Romantica Exhibition’ in Milan, Italy.