What is the connection between Utah Salt Lab and SecCodePLT?
Utah Salt Lab provides unit tests for the SecCodePLT benchmark. After filtering, 1,201 of the original 1,345 tasks have unit tests.
U
Utah Salt Lab MCP Server
@Utah-SaLT-Lab
基本信息
关于 Utah Salt Lab MCP Server
暂无概览
Utah Salt Lab MCP Server 配置
Utah Salt Lab MCP Server 工具
工具列表暂未提供
当前收录暂时没有可展示的工具列表，不代表服务没有工具。请查阅项目文档了解支持的能力。
工具是从 README 中自动提取的。维护者可以在 ## Tools 标题下列出工具,即可填充这部分内容。
概览
What is Utah Salt Lab?
Utah Salt Lab is a unit testing framework for evaluating code generated on the SecCodePLT benchmark. It provides containerized execution and preprocessing scripts to run unit tests on generated code and output evaluation results. It is designed for researchers studying LLM secure code generation.
How to use Utah Salt Lab?
Install dependencies via pip install -r requirements.txt, build a container using Docker or CharlieCloud, then run python preprocess.py to distribute generated code into test folders, execute sh run.sh, and finally run python get_result.py to collect evaluation results.
Key features of Utah Salt Lab
- Supports 1,201 SecCodePLT tasks with generated unit tests.
- Containerized environment recommended via Docker or Charliecloud.
- Preprocessing script distributes generated code into unit test folders.
- Results aggregation script outputs final evaluation metrics.
- Task IDs listed in
utils/SecPLT_func_name.json.
Use cases of Utah Salt Lab
- Evaluate secure code generation from large language models.
- Compare code quality across different LLMs on security tasks.
- Reproduce experiments from the associated arXiv paper (2503.15554).
常见问题
How do I set up the container environment?
Use the provided Dockerfile or CharlieCloud. Set `CH_IMAGE_STORAGE` to a directory with sufficient space, then build with `ch-image build -t test-runner -f Dockerfile .`.
Where are the evaluation results stored?
Results are saved via `get_result.py` to the path specified with `--output`. Intermediate files are stored in `data/` subdirectories.
How should I format generated code for testing?
Store code in a `.jsonl` file with each line containing `{"task_id": "<id>", "solution": "<code>"}`. The task IDs must be among the supported 1,201.
Why is containerized execution recommended?
The generated code may contain dangerous operations such as `rm -rf`. A container isolates the test environment and prevents unintended system damage.
其他 分类下的更多 MCP 服务器
Lemon.io
lemon-ioRequest dedicated senior developers to work on your project, write job descriptions, and prep technical interviews — all without leaving your Claude chat. Just describe what you need, and Lemon.io MCP does the rest. 1–3
🚀 Model Context Protocol (MCP) Curriculum for Beginners
microsoftThis open-source curriculum introduces the fundamentals of Model Context Protocol (MCP) through real-world, cross-language examples in .NET, Java, TypeScript, JavaScript, Rust and Python. Designed for developers, it focuses on practical techniques for building modular, scalable,
16.7K
Screenpipe
screenpipeYC (S26) | Open Computer History | Continuously record your company computer work, map your workflows, help you find work worth automating, and power your agents' context
22.0K
Senso MCP for Joomla
sensomediaSenso MCP turns a Joomla 5 or 6 site into a remote MCP server. It runs inside Joomla itself (no extra server, no Node.js bridge) and is secured with OAuth 2.1 and Joomla's own ACL. The free pack lets an AI assistant crea
Health by Dr Brad
Dr Brad StanfieldHealth by Dr Brad works out a health plan from your own results: which screening is due at your age, and evidence-based suggestions, each with the reason and the published guideline or study behind it. Your record is one
评论