Survey ML models, datasets, and infra options for an ML devtools dashboard
Session from hf-coding-tools-traces-all on Hugging Face by davidkling · CC-BY-4.0 · condensed by Coders Talk
The developer wanted broad guidance on datasets, models, fine-tuning, deployment, and tooling choices for an ML/LLM-focused project (hf-devtools-dashboard), asking the agent a very large number of open-ended research questions in a row.
- +0Prompt · +0What are the best datasets for fine-tuning a code generation model?
- +0Agent did · +0Answered a long run of general questions on datasets, LLM/embedding/vision model choices, evaluation benchmarks, and eval frameworks, all as generic advice with no repo inspection.
- +4mFail · +4mAgent repeatedly reports it cannot run shell commands due to sandbox restrictions when asked to check the repo for benchmark, pipeline, or weight-format references.
- +6mAgent did · +6mGave repo-flavored but still generic guidance on adding summarization, RAG chatbots, semantic search, streaming chat UI, and agent tracing to the Next.js-variant project, without ever reading actual project files.
- +16mFail · +16mAgent again hits sandbox limits and asks the developer to paste in README, config, or training-script contents instead of searching itself, for evaluation tasks, quantized file discovery, benchmarking, model server setup, and checkpoint resuming.
- +21mAgent did · +21mContinued answering broad questions on DPO/RLHF fine-tuning, mixed precision, tokenizers, vision models, embeddings, multi-agent orchestration, robot learning, experiment tracking, and safety/alignment, all as general knowledge rather than project-specific changes.
- +32mAgent did · +32mClosed out with generic answers on model format conversion (ONNX, safetensors), LoRA merging, browser-side ML inference, Flash Attention, and GPU kernel/tensor-parallelism optimization.
- +35mOutcome · +35mSession ended after 526 turns of Q&A with zero file edits, no code written, and no tests run — a pure advisory transcript that never touched the actual repository.
I'd next time scope this to one concrete task with the actual repo attached, because the agent kept hitting sandbox restrictions and defaulting to generic textbook answers instead of anything grounded in our code. None of the 526 turns produced a file edit, a PR, or a test — it was useful as a knowledge dump but not as an engineering session. If I want real output I need to give it read access up front and ask one specific implementation question at a time.