DocEmbedder is a class that embeds documents and chunks them into a format expected by the FeatureView schema using a Logic Implementation By the user.
Args:
repo_path: Path to the feature repo (can be "." for current directory).
yaml_file: Name of the feature_store.yaml file inside repo_path.
Defaults to "feature_store.yaml".
feature_view_name: Name of the feature view to create.
chunker: Chunker to use for chunking the documents.
embedder: Embedder to use for embedding the documents.
schema_transform_fn: Schema transform function to use for transforming the output of the chunker and embedder into the format expected by the FeatureView schema.
create_feature_view: Whether to create a feature view in the feature repo. By default it will generate a Python file with the FeatureView definition.
vector_length: Explicit embedding dimension for the generated FeatureView schema.
If None (default), the dimension is auto-detected from the embedder
via ``get_embedding_dim("text")``. Falls back to 384 if detection
is not supported by the embedder.
auto_apply_repo: Whether to apply the repository automatically. By default it will apply the repository after creating the feature view.
Apply the repository to register feature views in the registry.
"""
fromfeast.repo_configimportload_repo_config
fromfeast.repo_operationsimportapply_total
original_cwd=None
try:
original_cwd=os.getcwd()
repo_path=Path(self.repo_path).resolve()
config=load_repo_config(
repo_path=repo_path,
fs_yaml_file=Path(self.yaml_path),
)
apply_total(
repo_config=config,
repo_path=repo_path,
skip_source_validation=True,
)
finally:
iforiginal_cwdisnotNone:
os.chdir(original_cwd)
defembed_documents(
self,
documents: pd.DataFrame,
id_column: str,
source_column: str,
type_column: Optional[str] =None,
column_mapping: Optional[tuple[str, str]] =None,
custom_schema_transform_fn: Optional[
Callable[[pd.DataFrame], pd.DataFrame]
] =None,
) ->pd.DataFrame:
"""
Embed a list of documents and chunk them into a format expected by the FeatureView schema using a Logic Implementation By the user and save the DataFrame to the online store.
Args:
documents: DataFrame containing the documents to embed.
id_column: Column name containing the document IDs.
source_column: Column name containing the document sources.
type_column: Column name containing the document types.
column_mapping: Tuple mapping source columns to (modality, output column).
custom_schema_transform_fn: Custom schema transform function to use for transforming the output of the chunker and embedder into the format expected by the FeatureView schema.