【TF Tutorial】Chapter 1: Overview of TensorFlow

1. Introduction to TensorFlow and Keras

1.1 Overview of TensorFlow and Keras

TensorFlow:
An open-source machine learning framework developed by Google.
TensorFlow's flexible architecture allows you to deploy computation across a variety of platforms (CPUs, GPUs, TPUs).

Keras:
High-level neural networks API, written in Python.
It allows for easy and fast prototyping, and supports both convolutional networks and recurrent networks.

1.2 Installing TensorFlow

To start using TensorFlow, you'll need to install it.

pip install tensorflow

1.3 Understanding TensorFlow's Ecosystem

TensorFlow's ecosystem includes several libraries and tools to facilitate machine learning development:

TensorFlow Core: The low-level TensorFlow API.
Keras: High-level API for building and training models.
TensorFlow Datasets: Pre-built datasets for easy machine learning experimentation.
TensorFlow Hub: Pre-trained models for transfer learning.
TensorFlow Lite: Lightweight solution for mobile and embedded devices.
TensorFlow.js: TensorFlow for JavaScript, for use in web browsers and Node.js.
TensorBoard: Visualization toolkit for machine learning experimentation.

1.4 Import

Keras is included within TensorFlow as tf.keras.

import tensorflow as tf
from tensorflow.keras import layers, models

2. Dataset

2.1 Dataset

First, make the dataset.

import numpy as np
import tensorflow as tf

# Generate dummy data
num_samples = 1000
num_features = 20
num_classes = 3

x_data = np.random.random((num_samples, num_features))
y_data = np.random.randint(num_classes, size=(num_samples,))

# Convert labels to categorical (for classification)
y_data = tf.keras.utils.to_categorical(y_data, num_classes=num_classes)

# Split into training and testing sets
train_size = int(0.8 * num_samples)
x_train, x_test = x_data[:train_size], x_data[train_size:]
y_train, y_test = y_data[:train_size], y_data[train_size:]

# Create tf.data.Dataset objects
train_dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))
test_dataset = tf.data.Dataset.from_tensor_slices((x_test, y_test))

# Shuffle and batch the training data
batch_size = 32
train_dataset = train_dataset.shuffle(buffer_size=1024).batch(batch_size)
test_dataset = test_dataset.batch(batch_size)

This is an example of a way to make a dataset.
Briefly explanation:
tf.keras.utils.to_categorical: apply one-hot encoding to categorical data
from_tensor_slices: make dataset from array-like objects.
shuffle: shuffle the data by using buffer_size augment as pool size.
batch: make batch dataset with specified size.

example of from_tensor_slices how to work.

This is how to work of that function.

import tensorflow as tf
import numpy as np

# Create feature and label arrays
features = np.array([[1, 2], [3, 4], [5, 6], [7, 8], [9, 10]])
labels = np.array([0, 1, 1, 0, 0])

# Create a tf.data.Dataset from the feature and label arrays
dataset = tf.data.Dataset.from_tensor_slices((features, labels))

# Specify batch size
batch_size = 2
batched_dataset = dataset.batch(batch_size)

# Iterate through the batched dataset and print each batch
for batch in batched_dataset:
    features_batch, labels_batch = batch
    print("Features batch:")
    print(features_batch.numpy())
    print("Labels batch:")
    print(labels_batch.numpy())
    print()

・output

Features batch:
[[1 2]
 [3 4]]
Labels batch:
[0 1]

Features batch:
[[5 6]
 [7 8]]
Labels batch:
[1 0]

Features batch:
[[ 9 10]]
Labels batch:
[0]

3. Model

3.1 Define the model with 3 API

Define the model here, there are 3 ways to define the model.
・ Sequential API
・ Functional API
・ Subclassing API

・Sequential API

model = models.Sequential([
    layers.Flatten(input_shape=(28, 28)),  # Flatten the input
    layers.Dense(128, activation='relu'),  # Fully connected layer with ReLU activation
    layers.Dropout(0.2),                   # Dropout layer for regularization
    layers.Dense(10)                       # Output layer with 10 units (one for each digit)
])

# Same to above
# model = models.Sequential()
# model.add(layers.Flatten(input_shape=(28, 28)))
# model.add(layers.Dense(128, activation='relu'))
# model.add(layers.Dropout(0.2))
# model.add(layers.Dense(10))

・Functional API

from tensorflow.keras import Model, Input
from tensorflow.keras.layers import Flatten, Dense, Dropout

inputs = Input(shape=(28, 28))
x = Flatten()(inputs)
x = Dense(128, activation='relu')(x)
x = Dropout(0.2)(x)
outputs = Dense(10)(x)

model = Model(inputs=inputs, outputs=outputs)

・Subclassing API

import tensorflow as tf

class MyModel(tf.keras.Model):
    def __init__(self):
        super(MyModel, self).__init__()
        self.flatten = tf.keras.layers.Flatten()
        self.dense1 = tf.keras.layers.Dense(128, activation='relu')
        self.dropout = tf.keras.layers.Dropout(0.2)
        self.dense2 = tf.keras.layers.Dense(10)
    
    def call(self, inputs):
        x = self.flatten(inputs)
        x = self.dense1(x)
        x = self.dropout(x)
        return self.dense2(x)

model = MyModel()

When to use Sequential API:
・When building simple models that have a linear stack of layers.
・When each layer has exactly one input tensor and one output tensor.

When to use Functional API:
・When building complex models that require non-linear topology, such as multi-input models, multi-output models, or models with shared layers.
・When you need to manipulate multiple layers or outputs, such as branching networks.
・When working with models that include auxiliary outputs.

When to use Subclassing API:
・When you need to build highly customized or dynamic models.
・call: Define the forward pass of your model here. This method is called automatically when you use functions like fit, evaluate, and train_on_batch.

3.2 Compile the model

・Simple Compile
We sepcify the optimizer, loss type and metrics for validation.

model.compile(optimizer='adam',
                loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
              metrics=['accuracy'])

・Weighted sum loss(Multiple loss)

Functional API

import tensorflow as tf
import numpy as np

# define the input layer
inputs = tf.keras.Input(shape=(32,))
# define the shared layer
x = tf.keras.layers.Dense(64, activation='relu')(inputs)
# define multiple outputs
output1 = tf.keras.layers.Dense(10, name='output1')(x)
output2 = tf.keras.layers.Dense(1, name='output2')(x)

# define the model
model = tf.keras.Model(inputs=inputs, outputs=[output1, output2])
model.compile(
    optimizer='adam',
    loss={
        'output1': tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
        'output2': tf.keras.losses.MeanSquaredError()
    },
    loss_weights={
        'output1': 0.7,
        'output2': 0.3
    },
    metrics={
        'output1': ['accuracy'],
        'output2': ['mse']
    }
)

train_data = np.random.random((1000, 32))
train_labels1 = np.random.randint(10, size=(1000,))
train_labels2 = np.random.random((1000, 1))

train_labels = {
    'output1': train_labels1,
    'output2': train_labels2
}

# train the model
model.fit(train_data, train_labels, epochs=5)

new_data = np.random.random((10, 32))
predictions = model.predict(new_data)

print(predictions)

The keys in metric dictionary should match the names of the outputs or layers in your model.

Subclassing API

Similar to Functional API.

import tensorflow as tf
import numpy as np

class MyModel(tf.keras.Model):
    def __init__(self):
        super(MyModel, self).__init__()
        self.dense1 = tf.keras.layers.Dense(64, activation='relu')
        self.dense2 = tf.keras.layers.Dense(10)
        self.dense3 = tf.keras.layers.Dense(1)

    def call(self, inputs):
        x = self.dense1(inputs)
        output1 = self.dense2(x)
        output2 = self.dense3(x)
        return output1, output2

def custom_loss(y_true, y_pred):
    loss1 = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)(y_true[0], y_pred[0])
    loss2 = tf.keras.losses.MeanSquaredError()(y_true[1], y_pred[1])
    return 0.7 * loss1 + 0.3 * loss2

model = MyModel()
model.compile(
    optimizer='adam',
    loss=custom_loss,
    metrics={'dense2': ['accuracy'], 'dense3': ['mse']}
)

train_data = np.random.random((1000, 32))
train_labels1 = np.random.randint(10, size=(1000,))
train_labels2 = np.random.random((1000, 1))
train_labels = [train_labels1, train_labels2]

# train the model
model.fit(train_data, train_labels, epochs=5)

new_data = np.random.random((10, 32))
predictions = model.predict(new_data)

print(predictions)

The keys in metric dictionary should match the names of the outputs or layers in your model.

4. Training

It's so simple.

model.fit(train_dataset, epochs=10, validation_data=test_dataset)

4.1 Augmentation

We can define dataroader(work like dataset) for augmentation.
・Example

from tensorflow.keras.preprocessing.image import ImageDataGenerator

# Create an ImageDataGenerator object with data augmentation parameters
datagen = ImageDataGenerator(
    rotation_range=20,
    width_shift_range=0.2,
    height_shift_range=0.2,
    shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=True,
    fill_mode='nearest'
)

# Fit the generator to the training data
datagen.fit(x_train)

# Use the generator to train the model
model.fit(datagen.flow(x_train, y_train, batch_size=32),
          steps_per_epoch=len(x_train) // 32,
          epochs=5,
          validation_data=(x_test, y_test))

4.2 Callbacks

Callbacks are a way to add custom behavior to the training process. Keras provides several built-in callbacks such as ModelCheckpoint, EarlyStopping, ReduceLROnPlateau, and more.
・Example

from tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping

# Create a ModelCheckpoint callback to save the best model
checkpoint = ModelCheckpoint('best_model.h5', save_best_only=True, monitor='val_loss', mode='min')

# Create an EarlyStopping callback to stop training when a monitored metric has stopped improving
early_stopping = EarlyStopping(monitor='val_loss', patience=5, mode='min')

# Train the model with callbacks
model.fit(x_train, y_train,
          epochs=50,
          batch_size=32,
          validation_split=0.2,
          callbacks=[checkpoint, early_stopping])

5. Evaluation

Also simple.

model.evaluate(x_val, y_val, verbose=2)

6. Prediction

Also...
This gets the output that is specified when the model is defined.

predictions = model.predict(test_data)

7. Custom training loop(option)

We can use training loop instead of .fit for more detail settings.

import tensorflow as tf

# Define the loss function and optimizer
loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
optimizer = tf.keras.optimizers.Adam()

# Training loop
epochs = 5
batch_size = 32
train_dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train)).shuffle(10000).batch(batch_size)
test_dataset = tf.data.Dataset.from_tensor_slices((x_test, y_test)).batch(batch_size)

for epoch in range(epochs):
    print(f'Epoch {epoch + 1}/{epochs}')
    
    # Training step
    for step, (x_batch_train, y_batch_train) in enumerate(train_dataset):
        with tf.GradientTape() as tape:
            logits = model(x_batch_train, training=True)
            loss = loss_fn(y_batch_train, logits)
        
        gradients = tape.gradient(loss, model.trainable_variables)
        optimizer.apply_gradients(zip(gradients, model.trainable_variables))
        
        if step % 100 == 0:
            print(f'Step {step}: Loss = {loss.numpy()}')
    
    # Evaluation step
    test_loss = 0
    test_accuracy = tf.keras.metrics.SparseCategoricalAccuracy()
    for x_batch_test, y_batch_test in test_dataset:
        logits = model(x_batch_test, training=False)
        test_loss += loss_fn(y_batch_test, logits).numpy()
        test_accuracy.update_state(y_batch_test, logits)
    
    print(f'Test Loss: {test_loss / len(test_dataset)}')
    print(f'Test Accuracy: {test_accuracy.result().numpy()}')

8. .build and .summary(option)

tensorflow has .build method and .summary method.
Ill write a brief explanation of these because I could't understand these at first. build: define the model and input shape explicitly, and be needed for summary summary`: show the quick summary of model architecture.

I think the main function of .build is preparing to .summary. Also, these are not always necessary, so they are often not written.