dwrappr.dataset module

class dwrappr.dataset.DataPoint(x: ndarray, y: ndarray | None = None)

Bases: object

Represents a data point with associated x and optional y data arrays.

This class is designed to encapsulate a data point represented by Numpy arrays and optionally associated data. It validates inputs during initialization to ensure they are Numpy arrays. The class also supports saving itself to and loading from joblib files, with helper methods for these tasks.

Attributes:

x: np.ndarray

The primary data array, must be a Numpy array.

y: Optional[np.ndarray]

The secondary or optional data array, can be a Numpy array or None.

classmethod load(filepath: str) → DataPoint

Load a DataPoint object from a .joblib file.

This method reads a .joblib file from the given filepath, validates its extension, and loads the data to instantiate a DataPoint object.

Args:

filepath (str) : Path to the .joblib file to be loaded. The file must have a ‘.joblib’ extension.

Returns:

DataPoint : An instance of DataPoint created using the data in the file.

Raises:

ValueError : If the file does not have a ‘.joblib’ extension.

save(filepath: str) → None

Saves the object’s data to a specified file in Joblib format.

Raises an error if the specified file does not have a ‘.joblib’ extension. Uses the internal representation of the object’s data converted to a dictionary.

Args:

filepath (str) : The path to the file where the object’s data will be saved.

Raises:

ValueError : If the specified filepath does not end with ‘.joblib’.

x: ndarray

y: ndarray | None = None

class dwrappr.dataset.DataSet(datapoints: list[~dwrappr.dataset.DataPoint] = <factory>, dtypes: dict[str, str] = <factory>, meta: ~dwrappr.dataset.DataSetMeta = <factory>)

Bases: object

Represents a dataset consisting of data points, metadata, and associated attributes.

The DataSet class is designed to store and manipulate a collection of data points, along with metadata and data types for features and targets. It provides multiple methods and properties for retrieving subsets of the dataset, accessing features and targets in various formats (e.g., numpy array, pandas DataFrame, PyTorch tensor), and loading/saving datasets.

Attributes:

datapointsList[DataPoint]

A list of data point objects that make up the dataset.

dtypesdict[str, str]

A dictionary mapping column names to their data types.

metaDataSetMeta

Metadata object that contains information such as feature names, target names, and dataset name.

property as_df: DataFrame

Returns the dataset object as a pandas DataFrame.

This property converts the stored DataPoints into a DataFrame. It concatenates the x and y DataFrames along the columns axis and applies the stored data types to the resulting DataFrame before returning it.

Returns:

pd.DataFrame: A DataFrame representation of the stored DataPoints, with the stored data types applied.

datapoints: list[DataPoint]

dtypes: dict[str, str]

property feature_names: list[str]

Returns the names of features used in the metadata.

This property provides access to the feature names attribute present in the metadata object. It retrieves and returns the list of feature names.

Returns:

List[str]: The list of feature names.

Example:

>>> ds.feature_names
['feature']

classmethod from_dataframe(df: DataFrame, meta: DataSetMeta, check_df=True) → DataSet

Create a new DataSet instance from a given pandas DataFrame and metadata.

This method constructs a DataSet object from a DataFrame by extracting features and target values based on the provided metadata. It also ensures that the DataFrame aligns with the metadata specifications and performs a check if enabled. Additionally, the method captures data types of the specified feature and target columns for later retransformation from the DataSet to a DataFrame.

Args:

df (pd.DataFrame): The input DataFrame containing data structured according to the provided metadata. meta (DataSetMeta): Metadata object that specifies feature and target column names, among other dataset properties. check_df (bool, optional): An flag that determines whether to validate the DataFrame against the metadata. Default is True.

Returns:

DataSet: A new DataSet instance populated with DataPoint objects derived from the input DataFrame.

Raises:

No exceptions specified, as this section is not included as per the guidelines.

Examples:

>>> import pandas as pd
>>> from dwrappr import DataSet, DataSetMeta
>>> file_path_meta = r"dwrappr\examples\data\example_dataset_meta.json"
>>> meta = DataSetMeta.load(file_path_meta)
>>> file_path_data = r"dwrappr\examples\data\example_data.csv"
>>> df = pd.read_csv(file_path_data)
>>> ds = DataSet.from_dataframe(df = df, meta = meta)
>>> ds
DataSet(datapoints=[DataPoint(x=array([12]), y=array([0])), DataPoint(x=array([7]), y=array([1])), DataPoint(x=array([15]), y=array([0])), DataPoint(x=array([9]), y=array([1]))], dtypes={'feature': dtype('int64'), 'target': dtype('int64')}, meta=DataSetMeta(name='example_data', time_series='False', synthetic_data='True', feature_names=['feature'], target_names=['target'], origin=None, year=None, url=None, sector=None, target_type=None, description=None))
todo (jacob): Add example without json

classmethod from_list(features: list, meta: DataSetMeta, targets: list = None) → DataSet

Creates a DataSet object from given lists of features and targets along with a DataSetMeta instance.

Args:

features (list): A list containing the feature data, where each sub-list represents a row of feature values. meta (DataSetMeta): The metadata associated with the dataset, including feature and target names. targets (list, optional) : A list containing the target data, where each sub-list represents a row of target values. Defaults to None.

Returns:

DataSet: Returns an instance of the DataSet object created from the provided features, targets, and metadata.

static get_available_datasets_in_folder(path: str) → DataFrame

Gets available datasets from a specified folder and combines them into a single DataFrame.

Scans the folder to identify dataset metadata, retrieves the datasets, and concatenates them into one DataFrame.

Args:

path (str): The file path to the folder containing datasets.

Returns:

pd.DataFrame: A DataFrame containing the combined data from all datasets found in the folder.

classmethod load(filepath: str) → DataSet

Loads a dataset object.

This function loads a DataSet object from a file in .joblib format while reconstructing necessary components such as DataPoint and DataSetMeta objects. Assumes the file contains serialized elements suitable for creating a DataSet.

Args:

filepath (str): The path to the .joblib file from which the DataSet will be loaded.

Returns:

DataSet: A fully reconstructed DataSet instance based on the data provided in the file.

Raises:

ValueError: If the provided file does not have the .joblib extension.

meta: DataSetMeta

property name: str

Returns the name attribute of the meta property.

This property retrieves the name stored in the meta attribute. It does not accept any arguments and directly returns the name as a string.

Returns:

str: The name value associated with the meta attribute.

Example:

>>> ds.name
'example_data'

property num_datapoints: int

Returns the number of datapoints in the dataset.

This property calculates the total count of datapoints currently present and provides this information as an integer.

Returns:

int: The total number of datapoints in the dataset.

save(filepath: str, drop_meta_json: bool = True) → None

Saves the current object state to a specified file path, optionally excluding a meta JSON file. Ensures the file has the correct extension before saving.

Args:

filepath (str): The file path to save the object to. Must end with ‘.joblib’. drop_meta_json (bool): Whether to drop saving the meta JSON file. Defaults to True.

Raises:

ValueError: If the provided file path does not have a ‘.joblib’ extension.

Returns:

None

split_dataset(first_ds_size: float, shuffle: bool = True, random_state: int = 42, group_by_features: List[str] | None = None) → Tuple[DataSet, DataSet]

Splits the dataset into two subsets based on a specified ratio. The split can optionally group data points by specific feature values to ensure grouped subsets stay intact.

Args:

first_ds_size (float): Proportion of the dataset to assign to the first subset. Should be a value between 0 and 1. shuffle (bool, optional): Whether to shuffle the dataset or groups before splitting. Defaults to True. random_state (int, optional): Random seed for reproducibility of shuffling. Defaults to 42. group_by_features ([List[str]], optional): List of feature names to group data points by before splitting. If None, no grouping is applied. Defaults to None.

Returns:

Tuple[‘DataSet’, ‘DataSet’]: A tuple containing the two resulting datasets after the split.

Example:

>>> ds(0.5)
(DataSet(datapoints=[DataPoint(x=array([12]), y=array([0])), DataPoint(x=array([9]), y=array([1]))], dtypes={'feature': dtype('int64'), 'target': dtype('int64')}, meta=DataSetMeta(name='example_data', time_series='False', synthetic_data='True', feature_names=['feature'], target_names=['target'], origin=None, year=None, url=None, sector=None, target_type=None, description=None)),

(DataSet(datapoints=[DataPoint(x=array([7]), y=array([1])), DataPoint(x=array([15]), y=array([0]))], dtypes={‘feature’: dtype(‘int64’), ‘target’: dtype(‘int64’)}, meta=DataSetMeta(name=’example_data’, time_series=’False’, synthetic_data=’True’, feature_names=[‘feature’], target_names=[‘target’], origin=None, year=None, url=None, sector=None, target_type=None, description=None)))

property target_names: list[str]

Returns the list of target names specified in the metadata.

The method fetches and provides a list containing the target names which are stored in the meta attribute. The list represents the names or labels that correspond to target values in a dataset or similar context.

Returns:

list[str]: A list of target names.

Example:

>>> ds.target_names
['target']

property x: array

Returns the x-coordinates of all datapoints in the current object.

This property compiles a list of the x-values from all elements in the ‘datapoints’ attribute and returns them as a NumPy array. The returning array provides a structured format of the x-coordinates for further computations or manipulations.

Returns:

np.array: A NumPy array containing the x-coordinates of the datapoints in the object.

Example:

>>> ds.x
0,12
1,7
2,15
3,9

property x_as_df: DataFrame

Returns the x attribute as a pandas DataFrame.

Provides a property method to process and return the x attribute formatted as a pandas DataFrame with updated data types. The output DataFrame’s schema is adjusted according to the stored metadata and type definitions.

Returns:

pd.DataFrame: A pandas DataFrame created from the x attribute, with columns named according to meta.feature_names and updated data types based on the metadata settings.

property x_as_tensor: torch.Tensor

Returns the x attribute of the instance as a PyTorch tensor.

This property converts the x attribute to a PyTorch tensor of type torch.float32. It requires PyTorch to be installed in the environment.

Returns:

torch.Tensor: The x attribute of the instance converted to a tensor.

Raises:

ImportError: If PyTorch is not installed in the environment.

property y: ndarray | None

Returns the y values extracted from all datapoints as a NumPy array.

Y values correspond to the ‘y’ attribute of each datapoint in the list of datapoints. If no datapoints are present, it returns None.

Return:

Optional[np.ndarray]: A NumPy array of y values from datapoints, or None if no datapoints exist.

Example:

>>> ds.y
0,0
1,1
2,0
3,1

property y_as_df: DataFrame

Returns the target variable as a pandas DataFrame.

This property provides a DataFrame representation of the target variable with column names corresponding to the target_names attribute. It also ensures that the DataFrame’s data types are updated consistent with any pre-defined data type information.

Returns:

pd.DataFrame: The target variable represented as a pandas DataFrame with appropriately updated data types.

property y_as_tensor: torch.Tensor

Returns the attribute ‘y’ as a PyTorch tensor.

This property converts the ‘y’ attribute of the object into a PyTorch tensor with a data type of float32. It requires PyTorch to be installed, and will raise an ImportError if it is not available.

Returns:

torch.Tensor: The attribute ‘y’ represented as a tensor of type torch.float32.

Raises:

ImportError: If PyTorch library is not installed.

class dwrappr.dataset.DataSetMeta(name: str, time_series: bool, synthetic_data: bool, feature_names: ~typing.List[str], target_names: ~typing.List[str] = <factory>, origin: str = None, year: str = None, url: str = None, sector: str = None, target_type: str = None, description: str = None)

Bases: object

Represents metadata information for a dataset, including details about its attributes, usage, and related files.

This class is designed to store and manipulate metadata for datasets, providing an interface for converting metadata to a pandas DataFrame, loading metadata from JSON files and scanning directories for metadata files.

Attributes:

name: str

Name of the dataset.

time_series: bool

Indicates whether the dataset contains time series data.

synthetic_data: bool

Indicates whether the dataset contains synthetic data.

feature_names: List[str]

List of feature names in the dataset.

target_names: List[str]

List of target names in the dataset.

origin: str

Source or origin of the dataset.

year: str

Year associated with the dataset.

url: str

URL to access further information about the dataset.

sector: str

Sector to which the dataset belongs.

target_type: str

Type of the target variable (e.g., ‘classification’, ‘regression’).

description: str

Description or additional details about the dataset.

property as_df: DataFrame

Returns the available Metadata as a Dataframe.

This property Converts the object’s attributes into a pandas DataFrame. Lists in attributes are transformed into comma-separated strings for better readability.

Returns:

pd.DataFrame: A single-row DataFrame representing the object’s metadata and attributes.

description: str = None

feature_names: List[str]

classmethod load(filepath: str) → DataSetMeta

Loads an instance of DataSetMeta from a JSON file.

This class method reads a JSON file and initializes an instance of DataSetMeta using the contents of the file. If the file provided does not have a .json extension, a ValueError is raised.

Args:

filepath (str) :The path to the JSON file that contains the data needed to initialize a DataSetMeta instance.

Returns:

DataSetMeta: An instance of DataSetMeta initialized with the data from the JSON file.

Raises:

ValueError : If the file specified by ‘filepath’ does not have a “.json” extension.

Example:

>>> file_path_meta = r"dwrappr\examples\data\example_dataset_meta.json"
>>> meta = DataSetMeta.load(file_path_meta)
>>> meta
DataSetMeta(name='example_data', time_series='False', synthetic_data='True', feature_names=['feature'], target_names=['target'], origin=None, year=None, url=None, sector=None, target_type=None, description=None)

name: str

origin: str = None

save(filepath: str) → None

Saves the instance data to a specified JSON file.

The method ensures that the file has a ‘.json’ extension before attempting to save the instance data. If the extension is incorrect, a ValueError is raised. The instance is first converted to a dictionary representation and then written to the specified file path.

Args:

filepath (str) : The path to the file where the instance data will be saved. The file must have a ‘.json’ extension.

Returns:

None

Raises:

ValueError: Raised if the file does not have a ‘.json’ extension.

classmethod scan_for_meta(path: str, recursive: bool = True) → List[DataSetMeta]

Scans the directory for metadata and corresponding dataset objects.

This function scans a specified directory for metadata and associated dataset object files and returns a list of DataSetMeta instances. Files with extensions ‘.joblib’ and corresponding ‘_meta.json’ are paired, with unpaired files logged.

Args:

path (str): The root directory path to scan for metadata and dataset object files. recursive (bool, optional): Indicates whether subdirectories should also be scanned. Defaults to True.

Returns:

List[DataSetMeta]: A list containing DataSetMeta objects where both’.joblib’ dataset files and matching ‘_meta.json’ metadata files are found. If any of these files are missing a counterpart, a warning is logged.

Example:

>>> DataSetMeta.scan_for_meta(r"dwrappr\examples\data")
[DataSetMeta(name='example_data', time_series='False', synthetic_data='True', feature_names=['feature'], target_names=['target'], origin=None, year=None, url=None, sector=None, target_type=None, description=None)]

sector: str = None

synthetic_data: bool

target_names: List[str]

target_type: str = None

time_series: bool

url: str = None

year: str = None