You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.
Dismiss alert
# // Keep the booleans together to avoid misalignment during copy-by-value.
# bool vocab_only; // only load the vocabulary, no weights
# bool use_mmap; // use mmap if possible
# bool use_direct_io; // use direct io, takes precedence over use_mmap
# bool use_mlock; // force system to keep model in RAM
# bool check_tensors; // validate model tensor data
# bool use_extra_bufts; // use extra buffer types (used for weight repacking)
# bool no_host; // bypass host buffer allowing extra buffers to be used
# bool no_alloc; // only load metadata and simulate memory allocations
# };
classllama_model_params(ctypes.Structure):
"""Parameters for llama_model
Attributes:
devices (ctypes.Array[ggml_backend_dev_t]): NULL-terminated list of devices to use for offloading (if NULL, all available devices are used)
tensor_buft_overrides(llama_model_tensor_buft_override): NULL-terminated list of buffer types to use for tensors that match a pattern
n_gpu_layers (int): number of layers to store in VRAM, a negative value means all layers
split_mode (int): how to split the model across multiple GPUs
main_gpu (int): the GPU that is used for the entire model. main_gpu interpretation depends on split_mode: LLAMA_SPLIT_NONE: the GPU that is used for the entire model LLAMA_SPLIT_ROW: the GPU that is used for small tensors and intermediate results LLAMA_SPLIT_LAYER: ignored
tensor_split (ctypes.Array[ctypes.ctypes.c_float]): proportion of the model (layers or rows) to offload to each GPU, size: llama_max_devices()
progress_callback (llama_progress_callback): called with a progress value between 0.0 and 1.0. Pass NULL to disable. If the provided progress_callback returns true, model loading continues. If it returns false, model loading is immediately aborted.
progress_callback_user_data (ctypes.ctypes.c_void_p): context pointer passed to the progress callback
kv_overrides (ctypes.Array[llama_model_kv_override]): override key-value pairs of the model meta data
vocab_only (bool): only load the vocabulary, no weights
use_mmap (bool): use mmap if possible
use_direct_io(bool): use direct io, takes precedence over use_mmap
use_mlock (bool): force system to keep model in RAM
check_tensors (bool): validate model tensor data
use_extra_bufts (bool): use extra buffer types (used for weight repacking)
no_host (bool): bypass host buffer allowing extra buffers to be used
no_alloc (bool): only load metadata and simulate memory allocations"""