What is the connection between Utah Salt Lab and SecCodePLT?
Utah Salt Lab provides unit tests for the SecCodePLT benchmark. After filtering, 1,201 of the original 1,345 tasks have unit tests.
U
Utah Salt Lab MCP Server
@Utah-SaLT-Lab
Basic information
About Utah Salt Lab MCP Server
No overview available yet
Utah Salt Lab MCP Server config
No standard config provided
This server doesn't expose a parseable MCP config block in its README. See the repository for install instructions.
RepositoryUtah Salt Lab MCP Server tools
Tool list not available yet
This listing does not currently have a tool list to display. This does not mean the server has no tools; check its documentation for supported capabilities.
We auto-extract tools from the README. The maintainer can list them under a ## Tools heading to populate this section.
Overview
What is Utah Salt Lab?
Utah Salt Lab is a unit testing framework for evaluating code generated on the SecCodePLT benchmark. It provides containerized execution and preprocessing scripts to run unit tests on generated code and output evaluation results. It is designed for researchers studying LLM secure code generation.
How to use Utah Salt Lab?
Install dependencies via pip install -r requirements.txt, build a container using Docker or CharlieCloud, then run python preprocess.py to distribute generated code into test folders, execute sh run.sh, and finally run python get_result.py to collect evaluation results.
Key features of Utah Salt Lab
- Supports 1,201 SecCodePLT tasks with generated unit tests.
- Containerized environment recommended via Docker or Charliecloud.
- Preprocessing script distributes generated code into unit test folders.
- Results aggregation script outputs final evaluation metrics.
- Task IDs listed in
utils/SecPLT_func_name.json.
Use cases of Utah Salt Lab
- Evaluate secure code generation from large language models.
- Compare code quality across different LLMs on security tasks.
- Reproduce experiments from the associated arXiv paper (2503.15554).
Frequently asked questions
How do I set up the container environment?
Use the provided Dockerfile or CharlieCloud. Set `CH_IMAGE_STORAGE` to a directory with sufficient space, then build with `ch-image build -t test-runner -f Dockerfile .`.
Where are the evaluation results stored?
Results are saved via `get_result.py` to the path specified with `--output`. Intermediate files are stored in `data/` subdirectories.
How should I format generated code for testing?
Store code in a `.jsonl` file with each line containing `{"task_id": "<id>", "solution": "<code>"}`. The task IDs must be among the supported 1,201.
Why is containerized execution recommended?
The generated code may contain dangerous operations such as `rm -rf`. A container isolates the test environment and prevents unintended system damage.
More Other MCP servers
DeepMark
DeepMark helps teachers deliver rapid, consistent marking with meaningful feedback for every student — in a fraction of the time. What once took a week, now takes one free period.
Health by Dr Brad
Dr Brad StanfieldHealth by Dr Brad works out a health plan from your own results: which screening is due at your age, and evidence-based suggestions, each with the reason and the published guideline or study behind it. Your record is one
Maestro
mobile-dev-incPainless E2E Automation for Mobile and Web
14.8K
Sequential Thinking
modelcontextprotocolModel Context Protocol Servers
88.2K
Activepieces
activepiecesAI Agents & MCPs & AI Workflow Automation • (~400 MCP servers for AI agents) • AI Automation / AI Agent with MCPs • AI Workflows & AI Agents • MCPs for AI Agents
23.2K
Comments