cleverdoc.utils package

Submodules

cleverdoc.utils.auxiliary module

class cleverdoc.utils.auxiliary.BinaryT

Bases: int

class cleverdoc.utils.auxiliary.BoundDecimal(value: str, precision: int, scale: int)

Bases: Decimal

Custom data type that represents a decimal with a specific scale and precision.

class cleverdoc.utils.auxiliary.ByteT

Bases: int

class cleverdoc.utils.auxiliary.LongT

Bases: int

class cleverdoc.utils.auxiliary.ShortT

Bases: int

cleverdoc.utils.auxiliary.create_bound_decimal_type(precision: int, scale: int) → Type[BoundDecimal]

Factory method that creates a new BoundDecimal type with the specified precision and scale.

cleverdoc.utils.dataclass module

cleverdoc.utils.dataclass.apply_nullability(dtype: DataType, is_nullable: bool) → DataType

Returns a new PySpark DataType with the nullable flag set to the given value.

cleverdoc.utils.dataclass.get_spark_type(py_type: Type) → DataType

Creates a mapping from a python type to a pyspark data type :param py_type: :return:

cleverdoc.utils.dataclass.is_field_nullable(field_name: str, hints: dict) → bool

Returns True if the given field name is nullable, based on the type hint for the field in the given hints dictionary.

cleverdoc.utils.dataclass.is_optional_type(py_type: Type) → bool

Returns True if the given type is an Optional type.

cleverdoc.utils.dataclass.map_dataclass_to_struct(dataclass_type: Type) → StructType

Map a Python data class to a PySpark struct.

Parameters:

dataclass_type – The Python data class to be mapped.

Returns:

A PySpark struct that corresponds to the data class.

cleverdoc.utils.display_utils module

cleverdoc.utils.display_utils.get_column_type(df: DataFrame, column_name: str) → str

cleverdoc.utils.display_utils.myprint(ds, indent=0)

Go through all items in the dataset and print them with custom format

Modelled after Dataset._pretty_str()

cleverdoc.utils.display_utils.show_dicom(df, fields='content', limit=5, width=700, show_meta=True, only_meta=False, desc_width=60, limit_frame=5)

cleverdoc.utils.display_utils.show_image(image, width=600, show_meta=True, id=0)

cleverdoc.utils.display_utils.show_images(df, field='image', limit=5, width=600, show_meta=True)

class cleverdoc.utils.display_utils.show_pdf_file(pdf, size=(600, 500))

Bases: object

cleverdoc.utils.display_utils.split_bylen(item, maxlen)

Requires item to be sliceable (with __getitem__ defined)

Module contents

cleverdoc.utils.get_aws_version()

cleverdoc.utils.get_name(path, keep_subfolder_level=0)

cleverdoc.utils.get_name_udf(path, keep_subfolder_level=0)