Yet Another Orchestrator

I wanted to jot down some things I have learned while building out rubbish, a project built on firecracker vms running agents in a VM sandbox.

• Durable execution -- I resisted introducing a proper workflow engine into the project to learn the rough edges of having to write your own agent scheduler and executor. It was very fragile and was breaking easily. Using a project like Temporal now makes sense to me.
• Putting agents in non-standard envs -- When an agent starts in a highly restricted workspace and is not knowledgeable about the setup it is going to really struggle with how to run things, especially if it needs any kind of dockerization or virtualization that won't be directly available
• Secrets -- No one has solved this one satisfactorily yet. All the solutions I have seen do not work at a universal level, they rely on some kind of proxy to switch out placeholder secrets with actual secrets. The real answer here should be identity everywhere, but that solution is so hard to have everywhere that agents are going to be useful in an Enterprise.
• Context Bloat -- Working on a boring chatbot has taught me that too much information is also a danger for agents. Because of the way they explore files and start to pick up context along the way they do not read full documents. They read portions, and if one portion of what they read heavily depends on the context of the whole document, you are going to be entering the danger zone.
• Trickle tokens -- This is an idea that keeps coming up with our work. There are a ton of LLM requests that can be slower, I don't necessarily need a rapid response, and when I go into auto-research loop land, or just any other loop to solve a problem, I don't care how long it takes, I just want it to get done. I would love to find a place offering SOTA AI, but with heavily metered connection so you can't use more than X tokens in some amount of time, and you can choose when and how long to run each llm call I need to make. I don't know how possible this is with current architectures out there. Especially given the amount of caching that is being done in Claude sessions, that loss of common cache because each trickle is going to just eat up vram if things need to stay in memory for a long time to solve your problem.
• Small problems -- Smaller problems and more of them. It is generally accepted that give the LLMs a significantly thorough and thought out plan will give better results. One of the things that I look to do when I write these is to build from the basics up to the more advanced idea. I will have it do one small task at a time, most of the time this ends up being conversational after an initial planning phase, but when I want them to run for a long time unattended, the best results are always things that can be broken down into very small definite, and verifiable tasks.
• Steering hooks -- The eval numbers are very impressive from the AWS side, I am building out a copy of it for myself to learn what the edges are as well, and hope to achieve such adherence, but I'm excited to find out where the rough edges are, and why certain decisions get made in the creation of steering hooks.
• Self Eval Engine -- This has happened multiple times where people are collecting feedback from the bots, and humans using them, but we aren't at a place to spend the time to have the bot try and improve itself. It's something that I am really interested in. I would like to have the ear of anyone looking to potentially just let something like this loose and see what it does, but also what are the morals and ethics it will seem to abide by. It could be terrible, it could be great.