From 2017 to 2020, Kordspace contributed to open-source work around GPT-2 and GPT-3 model training, and provided early-access API services for those models through the ARCQ AI project.
This was before consumer chat products existed. Working with large language models meant collecting data, training and fine-tuning models by hand, and working through issues in public alongside the model maintainers.
Through ARCQ we offered developers early API access to GPT-2 and then GPT-3, so they could build on these models while they were still new.
The challenge
Early GPT models were new and hands-on. Training, testing and releasing task-specific models meant working through bugs and gaps in public, with little tooling. Serving the models to other developers also meant building gateways, keys, rate limits and metering from scratch, and running costly GPU workloads efficiently.
Our approach
We helped resolve issues on GitHub while building, testing and releasing public models trained on specific tasks. We ran the full machine-learning cycle of that era by hand: collecting and cleaning data, tokenising, training, checkpointing and evaluating. For ARCQ we built an API layer with keys, rate limits and usage metering in front of GPU training and inference in Python with TensorFlow and PyTorch.
The outcome
Task-trained models were released publicly and early API access was offered through ARCQ, giving developers a way to use GPT models before any consumer chat product existed. The tooling has since evolved far beyond those years, but the work built deep, practical experience with language models.
Technical highlights
- GPT-2 and GPT-3 open-source work: reproduced training runs and reported and fixed issues on GitHub alongside the maintainers.
- Task-specific fine-tuning: GPT-2 checkpoints tuned for specific jobs, then evaluated and published for public use.
- Full ML cycle by hand: data collection and cleaning, tokenization, training, checkpointing, perplexity and quality evaluation, and release.
- ARCQ AI: early-access API service for GPT-2 and then GPT-3, letting people use the models before any consumer chat product existed.
- Model serving: API gateway, keys, rate limits, usage metering and GPU training/inference runs in Python with TensorFlow and PyTorch.
- Community support: triaged issues, shared training recipes and answered questions for developers building on early models.
- Context: this reflects how model work was done in 2017–2020; the field has changed a great deal since.
Place in the journey
2017–2019 Software + CloudPart of the 2017–2019 Software + Cloud chapter of Marco Lopes' work.