llama.cpp/tests/CMakeLists.txt at e6c9e0986c79ba1cc8848879b2fcce979f9b4672 - aditya/llama.cpp - Forgejo: Beyond coding. We Forge.

aditya/llama.cpp

mirror of https://git.adityakumar.xyz/llama.cpp.git synced 2024-11-13 00:39:47 +00:00

Georgi Gerganov eb34620aec

Add tokenizer test + revert to C++11 (#355 )

* Add test-tokenizer-0 to do a few tokenizations - feel free to expand
* Added option to convert-pth-to-ggml.py script to dump just the vocabulary
* Added ./models/ggml-vocab.bin containing just LLaMA vocab data (used for tests)
* Added utility to load vocabulary file from previous point (temporary implementation)
* Avoid using std::string_view and drop back to C++11 (hope I didn't break something)
* Rename gpt_vocab -> llama_vocab
* All CMake binaries go into ./bin/ now

2023-03-21 17:29:41 +02:00

4 lines

257 B

CMake

Raw Blame History

 set(TEST_TARGET test-tokenizer-0)
 add_executable(${TEST_TARGET} ${TEST_TARGET}.cpp)
 target_link_libraries(${TEST_TARGET} PRIVATE utils)
 add_test(NAME ${TEST_TARGET} COMMAND $<TARGET_FILE:${TEST_TARGET}> ${CMAKE_CURRENT_SOURCE_DIR}/../models/ggml-vocab.bin)