Transformation

1# Dataset Transformation Template
2# Cell structure for a dataset transformation notebook.
3# The transformation function (Cell 2) is generated dynamically based on the user's
4# source and target formats. All other cells follow this skeleton.
5
6# Cell 0 [markdown]: Dataset Transformation
7# Description of the transformation (source format → target format)
8
9# Cell 1: Configuration
10
11INPUT_LOCATION = "[INPUT_LOCATION]"  # S3 URI or local path to input dataset
12OUTPUT_LOCATION = "[OUTPUT_LOCATION]"  # S3 URI or local path for output
13
14# Cell 2: Transformation Function
15# This cell is generated dynamically based on the user's source → target format.
16# In notebook mode, it uses %%writefile to save the function to transform_fn.py.
17# In script mode, the function is written to disk directly.
18# It must define:
19#
20#   def transform_dataset(df: pd.DataFrame) -> pd.DataFrame:
21#       ...
22#
23# The function should ONLY transform the DataFrame schema. No I/O, no side effects.
24
25# Cell 3: Load Dataset
26
27import pandas as pd
28from transform_fn import transform_dataset
29
30df = pd.read_json(INPUT_LOCATION, lines=True)
31print(f"Loaded {len(df)} records")
32print(f"Columns: {list(df.columns)}")
33df.head(2)
34
35# Cell 4: Transform
36
37df_transformed = transform_dataset(df)
38print(f"Transformed {len(df_transformed)} records")
39print(f"Columns: {list(df_transformed.columns)}")
40df_transformed.head(2)
41
42# Cell 5: Save Output
43
44df_transformed.to_json(OUTPUT_LOCATION, orient="records", lines=True)
45print(f"Saved {len(df_transformed)} records to {OUTPUT_LOCATION}")