| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
| Name | Name | Last commit date | ||
|---|---|---|---|---|
A flexible and powerful data generation tool built in Go that generates synthetic data based on YAML manifest definitions.
tables:
- name: users
priority: 1
columns:
- name: id
pattern: "USER####"
parent: true
validation:
unique: true
- name: name
type: string
value: ["John Doe", "Jane Doe"]
- name: metadata
type: json
json_config:
min_keys: 2
max_keys: 4
fields: ["age", "email", "preferences"]
types: ["int", "email", "string"]
- name: created_at
type: timestamp
format: "2006-01-02 15:04:05"go run main.go -manifest manifest/application.yaml -count 1000The Data Generator follows a modular architecture designed for flexibility and extensibility:
The generator supports Cassandra-specific data types for generating data that matches Cassandra's data model:
- name: user_preferences
type: map
key_type: string
value_type: string
map_config:
min_entries: 2
max_entries: 5
keys: ["theme", "language", "notifications"]
values: ["dark", "light", "en", "fr", "on", "off"]- name: tags
type: set
element_type: string
set_config:
min_elements: 1
max_elements: 3
values: ["urgent", "important", "normal", "low"]- name: address
type: udt
udt_config:
name: address_type
fields:
- name: street
type: string
- name: city
type: string
- name: state
type: string
- name: zip_code
type: string
pattern: "#####"- name: phone_numbers
type: list
element_type: string
list_config:
min_elements: 1
max_elements: 3
pattern: "+1-###-###-####"- name: coordinates
type: tuple
tuple_config:
elements:
- type: decimal
range:
min: -180.0
max: 180.0
- type: decimal
range:
min: -90.0
max: 90.0map_config:
min_entries: 1 # Minimum number of key-value pairs
max_entries: 5 # Maximum number of key-value pairs
keys: # Optional predefined keys
- key1
- key2
values: # Optional predefined values
- value1
- value2
key_type: string # Type of keys (string, int, etc.)
value_type: string # Type of values (string, int, etc.)set_config:
min_elements: 1 # Minimum number of elements
max_elements: 5 # Maximum number of elements
values: # Optional predefined values
- value1
- value2
element_type: string # Type of elementsudt_config:
name: type_name # Name of the UDT
fields: # List of fields in the UDT
- name: field1
type: string
- name: field2
type: intlist_config:
min_elements: 1 # Minimum number of elements
max_elements: 5 # Maximum number of elements
pattern: "pattern" # Optional pattern for elements
element_type: string # Type of elementstuple_config:
elements: # List of element configurations
- type: string # Type of first element
- type: int # Type of second element
range: # Optional range for numeric types
min: 1
max: 100tables:
- name: table_name # Table name
priority: 1 # Processing priority (higher numbers = higher priority)
depends_on: other_table # Table dependency
validation:
min_records: 1 # Minimum records to generate
max_records: 1000 # Maximum records to generatecolumns:
- name: column_name # Column name
type: string # Data type
pattern: "ABC####" # Pattern for generated values
value: ["A", "B"] # Predefined values
mandatory: true # Required field
validation:
unique: true # Unique constraint
range: # Value range
min: 1
max: 100
format: "format_string" # Format specificationcolumns:
- name: metadata
type: json
json_config:
min_keys: 2 # Minimum number of keys in JSON
max_keys: 5 # Maximum number of keys in JSON
fields: # Predefined field names
- name
- age
- email
types: # Corresponding field types
- string
- int
- emailThe data generator features a powerful rule-based data generation system with expressions. Rules can be defined at both column and table levels.
rules:
# Time-based rules
- when: "fields.submitted_date <= fields.created_on || addDuration(fields.created_on, '2h') > fields.submitted_date"
then:
submitted_date: "${addDuration(fields.created_on, '2h')}"
# Conditional value setting
- when: "fields.salary > 50000"
then:
priority: "HIGH"
otherwise:
priority: "${fields.salary > 25000 ? 'MEDIUM' : 'LOW'}"The expression engine provides a rich set of helper functions and variables in its evaluation environment:
Time Functions:
Math Functions:
String Functions:
- when: "fields.submitted_date <= fields.created_on"
then:
submitted_date: "${addDuration(fields.created_on, '2h')}"- when: 'fields.status == "COMPLETED"'
then:
completed_on: "${addDuration(fields.modified_on, '2h')}"- when: 'fields.status == "IN_PROGRESS" && fields.priority == "HIGH"'
then:
completed_on: "${addDuration(fields.modified_on, '1h')}"
modified_by: "John Doe"
otherwise:
completed_on: "${addDuration(fields.modified_on, '2h')}"
modified_by: "Jane Doe"Supported JSON field types:
- name: users
columns:
- name: id
pattern: "U####"
validation:
unique: true
- name: profile
type: json
json_config:
fields: ["age", "location", "interests"]
types: ["int", "string", "string"]- name: orders
depends_on: users
columns:
- name: order_id
pattern: "ORD####"
- name: user_id
foreign: "users.id"
- name: metadata
type: json
json_config:
fields: ["items", "total", "shipping"]The CSV sink allows you to output generated data to CSV files. Each table will be written to a separate CSV file in the specified output directory.
Example usage in your manifest:
sink:
type: csv
config:
output_dir: "./output"Features:
The CSV files will be named after the table names (e.g., users.csv, orders.csv). Each file will include a header row with column names followed by the data rows.
JSON fields are formatted in a readable string format: {key1:value1,key2:value2}.
The codebase follows a modular architecture for maintainability:
The expression evaluation environment is centralized in the initEnv function, which provides a consistent set of helper functions and variables for all expressions in the system. This ensures:
Contributions are welcome! Please feel free to submit a Pull Request.
| Back | FazBrowse Home | New Git URL |