⛓️🛠️ ChainForge
ChainForge is an open-source visual programming environment for prompt engineering, LLM evaluation and experimentation. With ChainForge, you can evaluate the robustness of prompts and text generation models with little to no coding required.
✨ Key Features
What You Can Do
- Query multiple LLMs at once to test prompt ideas and variations quickly and effectively. Or query the same LLM at different settings.
- Compare response quality across prompt permutations, across models, and across model settings to choose the best prompt and model for your use case.
- Setup evaluation metrics with code or LLM-based scorers and automatically plot results across prompts, prompt parameters, models, or model settings.
- Hold multiple conversations at once across template parameters and chat models. Template chat messages and inspect and evaluate outputs at each turn of a chat conversation.
Current Status
ChainForge is in active development and is provided as an open beta test.
🚀 How to Install
Install ChainForge locally for the best experience:
uv tool install chainforge # OR pip install chainforge
chainforge serve
Open localhost:8000 in a Chrome, Firefox, Edge or Brave browser.
For more details on installation, such as setting API keys as environment variables, see Getting Started.
Test ChainForge without installing anything:
Online Limitations
The web version has some limitations but includes the magical Share Button feature for sharing your experiments as links!
With Share, you can send your LLM experiments to others as links. Simply click Share to generate a unique weblink for your LLM experiment and copy it to your clipboard. For instance, here's an experiment that tries to get an LLM to reveal a secret key.
Browser Compatibility & API Keys
ChainForge is compatible with Google Chrome and Mozilla Firefox. You will need an API key for the LLM(s) you wish to use. We do not store your API keys —not in a cookie, localStorage, or server. Because of this, you must set your API keys every time you load ChainForge. If you prefer not to worry about it, we recommend installing ChainForge locally and setting your API keys as environment variables.
🌐 Join Our Community
Connect with other ChainForge users, get help, share your experiments, and stay updated with the latest developments!
👥 About Us
Meet the Team
ChainForge was created by Ian Arawjo during his time as Postdoctoral Fellow in the Variation Lab of the Harvard HCI group. He currently serves as the chief developer and maintainer.
Collaborators include:
- PhD students Priyan Vaithilingam and Chelse Swoopes
- Undergraduate Sean Yang
- Faculty members Elena Glassman and Martin Wattenberg
Special contributions:
- Chelse developed the documentation and helped with our user study
- Sean Yang pioneered the new Generate, Replace & Extend generative AI features
📚 How to Cite
If you use ChainForge for research purposes, or build upon the source code, we ask that you cite our arXiv pre-print in any related publications. The BibTeX to use is:
@misc{arawjo2023chainforge,
title={ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing},
author={Ian Arawjo and Chelse Swoopes and Priyan Vaithilingam and Martin Wattenberg and Elena Glassman},
year={2023},
eprint={2309.09128},
archivePrefix={arXiv},
primaryClass={cs.HC}
}