Tuning the hyperparameters of a neural network using EasyVVUQ
In this tutorial we will use the EasyVVUQ GridSampler to perform a grid search on the hyperparameters of a simple Keras neural network model, trained to recognize hand-written digits. This is the famous MNIST data set, of which 4 input features (of size 28 x 28) are show below. These are fed into a standard feed-forward neural network, which will predict the label 0-9.
Note: This tutorial always runs on the localhost. One possibility of performing the grid search on a remote supercomputer involves the use of FabSim, see the hyperparameter_tuning_tutorial_with_fabsim.ipynb notebook tutorial.
The (Keras) neural network script is located in mnist/keras_mnist.template, which will form the input template for the EasyVVUQ encoder. We will assume you are familiar with the basic EasyVVUQ building blocks. If not, you can look at the basic tutorial.
We need EasyVVUQ, TensorFlow and the TensorFlow data sets to execute this tutorial. If you need to install these, uncomment the corresponding line below.
[1]:
# !pip install easyvvuq
# !pip install tensorflow
# !pip install tensorflow_datasets
[2]:
import easyvvuq as uq
import os
import numpy as np
from easyvvuq.actions import CreateRunDirectory, Encode, Decode, ExecuteLocal, Actions
/Volumes/UserData/dpc/GIT/EasyVVUQ/env_3.12/lib/python3.12/site-packages/chaospy/__init__.py:9: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
import pkg_resources
We now set some flags:
[3]:
# Work directory, where the EasyVVUQ directory will be placed
WORK_DIR = '/tmp'
# target output filename generated by the code
TARGET_FILENAME = 'output.csv'
# EasyVVUQ campaign name
CAMPAIGN_NAME = 'grid_test'
As is standard in EasyVVUQ, we now define the parameter space. In this case these are 4 hyperparameters. There is one hidden layer with n_neurons neurons, a Dropout layer after the input and hidden layer, with dropout probability dropout_prob_in and dropout_prob_hidden respectively. We made the learning_rate tuneable as well.
[4]:
params = {}
params["n_neurons"] = {"type":"integer", "default": 32}
params["dropout_prob_in"] = {"type":"float", "default": 0.0}
params["dropout_prob_hidden"] = {"type":"float", "default": 0.0}
params["learning_rate"] = {"type":"float", "default": 0.001}
These 4 hyperparameter appear as flags in the input template mnist/keras_mnist.template. Typically this is generated from an input file used by some simulation code. In this case however, mnist/keras_mnist.template is directly our Python script, with the hyperparameters replaced by flags. For instance:
model = tf.keras.models.Sequential([
tf.keras.layers.Flatten(input_shape=(28, 28)),
tf.keras.layers.Dropout($dropout_prob_in),
tf.keras.layers.Dense($n_neurons, activation='relu'),
tf.keras.layers.Dropout($dropout_prob_hidden),
tf.keras.layers.Dense(10)
])
is simply the neural network construction part with flags for the dropout probabilities and the number of neurons in the hidden layer. The encoder reads the flags and replaces them with numeric values, and it subsequently writes the corresponding target_filename=hyper_param_tune.py:
[5]:
encoder = uq.encoders.GenericEncoder('./mnist/keras_mnist.template', target_filename='hyper_param_tune.py')
What follows are standard steps in setting up an EasyVVUQ Campaign
[6]:
# execute the runs locally
execute = ExecuteLocal('python3 hyper_param_tune.py')
# decode the output CSV files, with stored training and test accuracy values
output_columns = ["accuracy_train", "accuracy_test"]
decoder = uq.decoders.SimpleCSV(target_filename=TARGET_FILENAME, output_columns=output_columns)
# actions are 1) create run dirs, 2) encode input template, 3) execute runs, 4) decode output files
actions = Actions(CreateRunDirectory(root='/tmp', flatten=True), Encode(encoder), execute, Decode(decoder))
# create the EasyVVUQ main campaign object
campaign = uq.Campaign(
name=CAMPAIGN_NAME,
work_dir=WORK_DIR,
)
# add the param definitions and actions to the campaign
campaign.add_app(
name=CAMPAIGN_NAME,
params=params,
actions=actions
)
/Volumes/UserData/dpc/GIT/EasyVVUQ/env_3.12/lib/python3.12/site-packages/cerberus/validator.py:618: UserWarning: These types are defined both with a method and in the'types_mapping' property of this validator: {'integer'}
warn(
/Volumes/UserData/dpc/GIT/EasyVVUQ/env_3.12/lib/python3.12/site-packages/cerberus/validator.py:618: UserWarning: These types are defined both with a method and in the'types_mapping' property of this validator: {'integer'}
warn(
As with the uncertainty-quantification (UQ) samplers, the vary is used to select which of the params we actually vary. Unlike the UQ samplers we do not specify an input probability distribution. This being a grid search, we simply specify a list of values for each hyperparameter. Parameters not in vary, but with a flag in the template, will be given the default value specified in params.
[7]:
vary = {"n_neurons": [64, 128], "learning_rate": [0.005, 0.01, 0.015]}
Note: we are mixing integer and floats in the vary dict. Other data types (string, boolean) can also be used.
The vary dict is passed to the Grid_Sampler. As can be seen, it created a tensor product of all 1D points specified in vary. If a single tensor product is not useful (e.g. because it creates combinations of parameters that do not makes sense), you can also pass a list of different vary dicts. For even more flexibility you can also write the required parameter combinations to a CSV file, and pass it to the CSV_Sampler instead.
[8]:
# create an instance of the Grid Sampler
sampler = uq.sampling.Grid_Sampler(vary)
# Associate the sampler with the campaign
campaign.set_sampler(sampler)
# print the points
print("There are %d points:" % (sampler.n_samples()))
sampler.points
There are 6 points:
[8]:
[array([[64, 0.005],
[64, 0.01],
[64, 0.015],
[128, 0.005],
[128, 0.01],
[128, 0.015]], dtype=object)]
Run the actions (create directories with hyper_param_tune.py files in it)
[9]:
###############################
# execute the defined actions #
###############################
campaign.execute().collate()
/Volumes/UserData/dpc/GIT/EasyVVUQ/env_3.12/lib/python3.12/site-packages/keras/src/layers/reshaping/flatten.py:37: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
super().__init__(**kwargs)
/Volumes/UserData/dpc/GIT/EasyVVUQ/env_3.12/lib/python3.12/site-packages/keras/src/layers/reshaping/flatten.py:37: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
super().__init__(**kwargs)
/Volumes/UserData/dpc/GIT/EasyVVUQ/env_3.12/lib/python3.12/site-packages/keras/src/layers/reshaping/flatten.py:37: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
super().__init__(**kwargs)
/Volumes/UserData/dpc/GIT/EasyVVUQ/env_3.12/lib/python3.12/site-packages/keras/src/layers/reshaping/flatten.py:37: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
super().__init__(**kwargs)
/Volumes/UserData/dpc/GIT/EasyVVUQ/env_3.12/lib/python3.12/site-packages/keras/src/layers/reshaping/flatten.py:37: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
super().__init__(**kwargs)
/Volumes/UserData/dpc/GIT/EasyVVUQ/env_3.12/lib/python3.12/site-packages/keras/src/layers/reshaping/flatten.py:37: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
super().__init__(**kwargs)
2025-07-20 09:15:35.505420: I tensorflow/core/kernels/data/tf_record_dataset_op.cc:387] The default buffer size is 262144, which is overridden by the user specified `buffer_size` of 8388608
2025-07-20 09:15:35.505447: I tensorflow/core/kernels/data/tf_record_dataset_op.cc:387] The default buffer size is 262144, which is overridden by the user specified `buffer_size` of 8388608
2025-07-20 09:15:35.505539: I tensorflow/core/kernels/data/tf_record_dataset_op.cc:387] The default buffer size is 262144, which is overridden by the user specified `buffer_size` of 8388608
2025-07-20 09:15:35.505632: I tensorflow/core/kernels/data/tf_record_dataset_op.cc:387] The default buffer size is 262144, which is overridden by the user specified `buffer_size` of 8388608
2025-07-20 09:15:35.505885: I tensorflow/core/kernels/data/tf_record_dataset_op.cc:387] The default buffer size is 262144, which is overridden by the user specified `buffer_size` of 8388608
2025-07-20 09:15:35.505924: I tensorflow/core/kernels/data/tf_record_dataset_op.cc:387] The default buffer size is 262144, which is overridden by the user specified `buffer_size` of 8388608
Epoch 1/6
Epoch 1/6
Epoch 1/6
Epoch 1/6
Epoch 1/6
Epoch 1/6
469/469 ━━━━━━━━━━━━━━━━━━━━ 5s 2ms/step - loss: 0.6675 - sparse_categorical_accuracy: 0.8001 - val_loss: 0.2092 - val_sparse_categorical_accuracy: 0.9369
Epoch 2/6
469/469 ━━━━━━━━━━━━━━━━━━━━ 5s 2ms/step - loss: 0.8846 - sparse_categorical_accuracy: 0.7448 - val_loss: 0.2860 - val_sparse_categorical_accuracy: 0.9189
Epoch 2/6
469/469 ━━━━━━━━━━━━━━━━━━━━ 6s 2ms/step - loss: 0.9380 - sparse_categorical_accuracy: 0.7193 - val_loss: 0.2912 - val_sparse_categorical_accuracy: 0.9184
Epoch 2/6
469/469 ━━━━━━━━━━━━━━━━━━━━ 6s 2ms/step - loss: 0.7561 - sparse_categorical_accuracy: 0.7718 - val_loss: 0.2439 - val_sparse_categorical_accuracy: 0.9297
Epoch 2/6
469/469 ━━━━━━━━━━━━━━━━━━━━ 6s 2ms/step - loss: 0.6991 - sparse_categorical_accuracy: 0.7860 - val_loss: 0.2195 - val_sparse_categorical_accuracy: 0.9373
Epoch 2/6
469/469 ━━━━━━━━━━━━━━━━━━━━ 6s 3ms/step - loss: 0.7696 - sparse_categorical_accuracy: 0.7651 - val_loss: 0.2370 - val_sparse_categorical_accuracy: 0.9342
Epoch 2/6
469/469 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.2755 - sparse_categorical_accuracy: 0.9221 - val_loss: 0.2228 - val_sparse_categorical_accuracy: 0.9370
469/469 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.1879 - sparse_categorical_accuracy: 0.9460 - val_loss: 0.1433 - val_sparse_categorical_accuracy: 0.9592
Epoch 3/6
Epoch 3/6
469/469 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step - loss: 0.2837 - sparse_categorical_accuracy: 0.9185 - val_loss: 0.2398 - val_sparse_categorical_accuracy: 0.9343
Epoch 3/6
469/469 ━━━━━━━━━━━━━━━━━━━━ 0s 986us/step - loss: 0.2392 - sparse_categorical_accuracy: 0.9322 - val_loss: 0.1889 - val_sparse_categorical_accuracy: 0.9437
Epoch 3/6
469/469 ━━━━━━━━━━━━━━━━━━━━ 0s 920us/step - loss: 0.2052 - sparse_categorical_accuracy: 0.9416 - val_loss: 0.1515 - val_sparse_categorical_accuracy: 0.9555
Epoch 3/6
469/469 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step - loss: 0.2255 - sparse_categorical_accuracy: 0.9354 - val_loss: 0.1763 - val_sparse_categorical_accuracy: 0.9501
Epoch 3/6
469/469 ━━━━━━━━━━━━━━━━━━━━ 0s 913us/step - loss: 0.2342 - sparse_categorical_accuracy: 0.9331 - val_loss: 0.2041 - val_sparse_categorical_accuracy: 0.9422
Epoch 4/6
469/469 ━━━━━━━━━━━━━━━━━━━━ 0s 916us/step - loss: 0.1874 - sparse_categorical_accuracy: 0.9467 - val_loss: 0.1601 - val_sparse_categorical_accuracy: 0.9516
Epoch 4/6
469/469 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step - loss: 0.2246 - sparse_categorical_accuracy: 0.9360 - val_loss: 0.1888 - val_sparse_categorical_accuracy: 0.9465
Epoch 4/6
469/469 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step - loss: 0.1337 - sparse_categorical_accuracy: 0.9614 - val_loss: 0.1111 - val_sparse_categorical_accuracy: 0.9661
Epoch 4/6
469/469 ━━━━━━━━━━━━━━━━━━━━ 0s 874us/step - loss: 0.1432 - sparse_categorical_accuracy: 0.9598 - val_loss: 0.1368 - val_sparse_categorical_accuracy: 0.9603
Epoch 4/6
469/469 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step - loss: 0.1595 - sparse_categorical_accuracy: 0.9538 - val_loss: 0.1346 - val_sparse_categorical_accuracy: 0.9616
Epoch 4/6
469/469 ━━━━━━━━━━━━━━━━━━━━ 0s 909us/step - loss: 0.1967 - sparse_categorical_accuracy: 0.9446 - val_loss: 0.1798 - val_sparse_categorical_accuracy: 0.9478
Epoch 5/6
469/469 ━━━━━━━━━━━━━━━━━━━━ 0s 893us/step - loss: 0.1515 - sparse_categorical_accuracy: 0.9558 - val_loss: 0.1423 - val_sparse_categorical_accuracy: 0.9584
Epoch 5/6
469/469 ━━━━━━━━━━━━━━━━━━━━ 0s 949us/step - loss: 0.1171 - sparse_categorical_accuracy: 0.9663 - val_loss: 0.1099 - val_sparse_categorical_accuracy: 0.9675
Epoch 5/6
469/469 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step - loss: 0.1825 - sparse_categorical_accuracy: 0.9486 - val_loss: 0.1656 - val_sparse_categorical_accuracy: 0.9511
Epoch 5/6
469/469 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step - loss: 0.0994 - sparse_categorical_accuracy: 0.9711 - val_loss: 0.1015 - val_sparse_categorical_accuracy: 0.9693
Epoch 5/6
469/469 ━━━━━━━━━━━━━━━━━━━━ 0s 928us/step - loss: 0.1793 - sparse_categorical_accuracy: 0.9485 - val_loss: 0.1654 - val_sparse_categorical_accuracy: 0.9513
Epoch 6/6
469/469 ━━━━━━━━━━━━━━━━━━━━ 0s 919us/step - loss: 0.1324 - sparse_categorical_accuracy: 0.9613 - val_loss: 0.1254 - val_sparse_categorical_accuracy: 0.9619
Epoch 6/6
469/469 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step - loss: 0.1268 - sparse_categorical_accuracy: 0.9644 - val_loss: 0.1231 - val_sparse_categorical_accuracy: 0.9645
Epoch 5/6
469/469 ━━━━━━━━━━━━━━━━━━━━ 0s 917us/step - loss: 0.0954 - sparse_categorical_accuracy: 0.9726 - val_loss: 0.1015 - val_sparse_categorical_accuracy: 0.9695
Epoch 6/6
469/469 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step - loss: 0.1617 - sparse_categorical_accuracy: 0.9549 - val_loss: 0.1515 - val_sparse_categorical_accuracy: 0.9555
Epoch 6/6
469/469 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step - loss: 0.0806 - sparse_categorical_accuracy: 0.9774 - val_loss: 0.0975 - val_sparse_categorical_accuracy: 0.9709
Epoch 6/6
469/469 ━━━━━━━━━━━━━━━━━━━━ 0s 909us/step - loss: 0.1600 - sparse_categorical_accuracy: 0.9543 - val_loss: 0.1473 - val_sparse_categorical_accuracy: 0.9577
469/469 ━━━━━━━━━━━━━━━━━━━━ 0s 936us/step - loss: 0.1134 - sparse_categorical_accuracy: 0.9673 - val_loss: 0.1163 - val_sparse_categorical_accuracy: 0.9636
469/469 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step - loss: 0.1063 - sparse_categorical_accuracy: 0.9695 - val_loss: 0.1035 - val_sparse_categorical_accuracy: 0.9696
Epoch 6/6
469/469 ━━━━━━━━━━━━━━━━━━━━ 0s 897us/step - loss: 0.0829 - sparse_categorical_accuracy: 0.9761 - val_loss: 0.1022 - val_sparse_categorical_accuracy: 0.9682
469/469 ━━━━━━━━━━━━━━━━━━━━ 0s 564us/step - loss: 0.1442 - sparse_categorical_accuracy: 0.9584
469/469 ━━━━━━━━━━━━━━━━━━━━ 0s 575us/step - loss: 0.1023 - sparse_categorical_accuracy: 0.9703
79/79 ━━━━━━━━━━━━━━━━━━━━ 0s 579us/step - loss: 0.1455 - sparse_categorical_accuracy: 0.9584
79/79 ━━━━━━━━━━━━━━━━━━━━ 0s 573us/step - loss: 0.1154 - sparse_categorical_accuracy: 0.9640
469/469 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step - loss: 0.1421 - sparse_categorical_accuracy: 0.9608 - val_loss: 0.1366 - val_sparse_categorical_accuracy: 0.9597
469/469 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step - loss: 0.0748 - sparse_categorical_accuracy: 0.9787 - val_loss: 0.0862 - val_sparse_categorical_accuracy: 0.9736
469/469 ━━━━━━━━━━━━━━━━━━━━ 0s 551us/step - loss: 0.0812 - sparse_categorical_accuracy: 0.9755
79/79 ━━━━━━━━━━━━━━━━━━━━ 0s 454us/step - loss: 0.0996 - sparse_categorical_accuracy: 0.9687
469/469 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 0.0885 - sparse_categorical_accuracy: 0.9751 - val_loss: 0.0929 - val_sparse_categorical_accuracy: 0.9726
469/469 ━━━━━━━━━━━━━━━━━━━━ 0s 550us/step - loss: 0.1273 - sparse_categorical_accuracy: 0.9654
469/469 ━━━━━━━━━━━━━━━━━━━━ 0s 556us/step - loss: 0.0601 - sparse_categorical_accuracy: 0.9832
79/79 ━━━━━━━━━━━━━━━━━━━━ 0s 501us/step - loss: 0.1349 - sparse_categorical_accuracy: 0.9615
79/79 ━━━━━━━━━━━━━━━━━━━━ 0s 477us/step - loss: 0.0868 - sparse_categorical_accuracy: 0.9715
469/469 ━━━━━━━━━━━━━━━━━━━━ 0s 466us/step - loss: 0.0764 - sparse_categorical_accuracy: 0.9796
79/79 ━━━━━━━━━━━━━━━━━━━━ 0s 403us/step - loss: 0.0914 - sparse_categorical_accuracy: 0.9721
Epoch 1/6
Epoch 1/6
Epoch 1/6
Epoch 1/6
Epoch 1/6
Epoch 1/6
469/469 [==============================] - 15s 13ms/step - loss: 0.4033 - sparse_categorical_accuracy: 0.8792 - val_loss: 0.2163 - val_sparse_categorical_accuracy: 0.9375
Epoch 2/6
469/469 [==============================] - 15s 9ms/step - loss: 0.4464 - sparse_categorical_accuracy: 0.8710 - val_loss: 0.2590 - val_sparse_categorical_accuracy: 0.9264
Epoch 2/6
469/469 [==============================] - 15s 13ms/step - loss: 0.5436 - sparse_categorical_accuracy: 0.8448 - val_loss: 0.2959 - val_sparse_categorical_accuracy: 0.9169
469/469 [==============================] - 16s 14ms/step - loss: 0.4242 - sparse_categorical_accuracy: 0.8773 - val_loss: 0.2251 - val_sparse_categorical_accuracy: 0.9334
Epoch 2/6
469/469 [==============================] - 16s 14ms/step - loss: 0.3825 - sparse_categorical_accuracy: 0.8884 - val_loss: 0.2041 - val_sparse_categorical_accuracy: 0.9407
Epoch 2/6
469/469 [==============================] - 16s 14ms/step - loss: 0.5121 - sparse_categorical_accuracy: 0.8543 - val_loss: 0.2868 - val_sparse_categorical_accuracy: 0.9208
296/469 [=================>............] - ETA: 0s - loss: 0.2019 - sparse_categorical_accuracy: 0.9404Epoch 2/6
469/469 [==============================] - 2s 5ms/step - loss: 0.2190 - sparse_categorical_accuracy: 0.9385 - val_loss: 0.1947 - val_sparse_categorical_accuracy: 0.9430
243/469 [==============>...............] - ETA: 1s - loss: 0.2207 - sparse_categorical_accuracy: 0.9359Epoch 3/6
469/469 [==============================] - 3s 6ms/step - loss: 0.1926 - sparse_categorical_accuracy: 0.9432 - val_loss: 0.1692 - val_sparse_categorical_accuracy: 0.9502
Epoch 3/6
469/469 [==============================] - 4s 8ms/step - loss: 0.2056 - sparse_categorical_accuracy: 0.9409 - val_loss: 0.1634 - val_sparse_categorical_accuracy: 0.9507
Epoch 3/6
469/469 [==============================] - 4s 8ms/step - loss: 0.1765 - sparse_categorical_accuracy: 0.9489 - val_loss: 0.1405 - val_sparse_categorical_accuracy: 0.9589
469/469 [==============================] - 4s 9ms/step - loss: 0.2690 - sparse_categorical_accuracy: 0.9233 - val_loss: 0.2342 - val_sparse_categorical_accuracy: 0.9324
Epoch 3/6
469/469 [==============================] - 3s 7ms/step - loss: 0.1725 - sparse_categorical_accuracy: 0.9514 - val_loss: 0.1574 - val_sparse_categorical_accuracy: 0.9550
84/469 [====>.........................] - ETA: 2s - loss: 0.2292 - sparse_categorical_accuracy: 0.9323Epoch 4/6
469/469 [==============================] - 3s 7ms/step - loss: 0.1440 - sparse_categorical_accuracy: 0.9579 - val_loss: 0.1253 - val_sparse_categorical_accuracy: 0.9631
Epoch 4/6
221/469 [=============>................] - ETA: 1s - loss: 0.1635 - sparse_categorical_accuracy: 0.9525Epoch 3/6
113/469 [======>.......................] - ETA: 1s - loss: 0.1428 - sparse_categorical_accuracy: 0.9583Epoch 2/6
469/469 [==============================] - 4s 8ms/step - loss: 0.1552 - sparse_categorical_accuracy: 0.9557 - val_loss: 0.1408 - val_sparse_categorical_accuracy: 0.9578
455/469 [============================>.] - ETA: 0s - loss: 0.1411 - sparse_categorical_accuracy: 0.9602Epoch 4/6
469/469 [==============================] - 3s 7ms/step - loss: 0.2198 - sparse_categorical_accuracy: 0.9381 - val_loss: 0.1971 - val_sparse_categorical_accuracy: 0.9432
469/469 [==============================] - 3s 6ms/step - loss: 0.1407 - sparse_categorical_accuracy: 0.9603 - val_loss: 0.1350 - val_sparse_categorical_accuracy: 0.9623
469/469 [==============================] - 3s 6ms/step - loss: 0.1177 - sparse_categorical_accuracy: 0.9659 - val_loss: 0.1115 - val_sparse_categorical_accuracy: 0.9675
437/469 [==========================>...] - ETA: 0s - loss: 0.2782 - sparse_categorical_accuracy: 0.9216Epoch 5/6
469/469 [==============================] - 3s 7ms/step - loss: 0.1296 - sparse_categorical_accuracy: 0.9632 - val_loss: 0.1169 - val_sparse_categorical_accuracy: 0.9652
58/469 [==>...........................] - ETA: 1s - loss: 0.1079 - sparse_categorical_accuracy: 0.9674Epoch 4/6
469/469 [==============================] - 3s 6ms/step - loss: 0.2760 - sparse_categorical_accuracy: 0.9222 - val_loss: 0.2362 - val_sparse_categorical_accuracy: 0.9323
361/469 [======================>.......] - ETA: 0s - loss: 0.1000 - sparse_categorical_accuracy: 0.9704Epoch 4/6
469/469 [==============================] - 2s 5ms/step - loss: 0.1241 - sparse_categorical_accuracy: 0.9643 - val_loss: 0.1146 - val_sparse_categorical_accuracy: 0.9661
Epoch 5/6
109/469 [=====>........................] - ETA: 1s - loss: 0.2055 - sparse_categorical_accuracy: 0.9414Epoch 5/6
469/469 [==============================] - 2s 4ms/step - loss: 0.1003 - sparse_categorical_accuracy: 0.9705 - val_loss: 0.1062 - val_sparse_categorical_accuracy: 0.9665
147/469 [========>.....................] - ETA: 1s - loss: 0.1988 - sparse_categorical_accuracy: 0.9439Epoch 6/6
119/469 [======>.......................] - ETA: 1s - loss: 0.1241 - sparse_categorical_accuracy: 0.9641Epoch 3/6
469/469 [==============================] - 3s 6ms/step - loss: 0.1030 - sparse_categorical_accuracy: 0.9703 - val_loss: 0.1046 - val_sparse_categorical_accuracy: 0.9687
67/469 [===>..........................] - ETA: 2s - loss: 0.2337 - sparse_categorical_accuracy: 0.9356Epoch 5/6
469/469 [==============================] - 3s 7ms/step - loss: 0.1873 - sparse_categorical_accuracy: 0.9469 - val_loss: 0.1769 - val_sparse_categorical_accuracy: 0.9504
248/469 [==============>...............] - ETA: 1s - loss: 0.0810 - sparse_categorical_accuracy: 0.9773Epoch 5/6
469/469 [==============================] - 3s 6ms/step - loss: 0.1213 - sparse_categorical_accuracy: 0.9653 - val_loss: 0.1245 - val_sparse_categorical_accuracy: 0.9628
Epoch 6/6
469/469 [==============================] - 3s 6ms/step - loss: 0.0867 - sparse_categorical_accuracy: 0.9755 - val_loss: 0.0991 - val_sparse_categorical_accuracy: 0.9702
469/469 [==============================] - 3s 7ms/step - loss: 0.1038 - sparse_categorical_accuracy: 0.9709 - val_loss: 0.1101 - val_sparse_categorical_accuracy: 0.9670
469/469 [==============================] - 3s 6ms/step - loss: 0.2256 - sparse_categorical_accuracy: 0.9364 - val_loss: 0.2024 - val_sparse_categorical_accuracy: 0.9425
469/469 [==============================] - 3s 6ms/step - loss: 0.0848 - sparse_categorical_accuracy: 0.9758 - val_loss: 0.0951 - val_sparse_categorical_accuracy: 0.9703
469/469 [==============================] - 2s 3ms/step - loss: 0.0748 - sparse_categorical_accuracy: 0.9791
469/469 [==============================] - 2s 4ms/step - loss: 0.1070 - sparse_categorical_accuracy: 0.9691 - val_loss: 0.1089 - val_sparse_categorical_accuracy: 0.9682
Epoch 6/6
469/469 [==============================] - 2s 5ms/step - loss: 0.1628 - sparse_categorical_accuracy: 0.9542 - val_loss: 0.1559 - val_sparse_categorical_accuracy: 0.9545
Epoch 6/6
311/469 [==================>...........] - ETA: 0s - loss: 0.0950 - sparse_categorical_accuracy: 0.9729Epoch 4/6
79/79 [==============================] - 0s 3ms/step - loss: 0.0991 - sparse_categorical_accuracy: 0.9702
84/469 [====>.........................] - ETA: 1s - loss: 0.2051 - sparse_categorical_accuracy: 0.9418Epoch 6/6
469/469 [==============================] - 1s 3ms/step - loss: 0.0933 - sparse_categorical_accuracy: 0.9735
469/469 [==============================] - 3s 5ms/step - loss: 0.0900 - sparse_categorical_accuracy: 0.9743 - val_loss: 0.0938 - val_sparse_categorical_accuracy: 0.9720
469/469 [==============================] - 3s 5ms/step - loss: 0.1441 - sparse_categorical_accuracy: 0.9596 - val_loss: 0.1390 - val_sparse_categorical_accuracy: 0.9595
79/79 [==============================] - 0s 3ms/step - loss: 0.1089 - sparse_categorical_accuracy: 0.9682
469/469 [==============================] - 2s 5ms/step - loss: 0.1951 - sparse_categorical_accuracy: 0.9449 - val_loss: 0.1784 - val_sparse_categorical_accuracy: 0.9480
Epoch 5/6
469/469 [==============================] - 3s 5ms/step - loss: 0.0724 - sparse_categorical_accuracy: 0.9797 - val_loss: 0.0847 - val_sparse_categorical_accuracy: 0.9726
469/469 [==============================] - 2s 3ms/step - loss: 0.1309 - sparse_categorical_accuracy: 0.9635
79/79 [==============================] - 0s 3ms/step - loss: 0.1390 - sparse_categorical_accuracy: 0.9595
460/469 [============================>.] - ETA: 0s - loss: 0.1721 - sparse_categorical_accuracy: 0.9512
[10]:
data_frame = campaign.get_collation_result()
data_frame
[10]:
| run_id | iteration | n_neurons | learning_rate | dropout_prob_in | dropout_prob_hidden | accuracy_train | accuracy_test | |
|---|---|---|---|---|---|---|---|---|
| 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |
| 0 | 1 | 0 | 64 | 0.005 | 0.0 | 0.0 | 0.959450 | 0.9577 |
| 1 | 2 | 0 | 64 | 0.010 | 0.0 | 0.0 | 0.970083 | 0.9636 |
| 2 | 3 | 0 | 64 | 0.015 | 0.0 | 0.0 | 0.976750 | 0.9682 |
| 3 | 4 | 0 | 128 | 0.005 | 0.0 | 0.0 | 0.965550 | 0.9597 |
| 4 | 5 | 0 | 128 | 0.010 | 0.0 | 0.0 | 0.978967 | 0.9726 |
| 5 | 6 | 0 | 128 | 0.015 | 0.0 | 0.0 | 0.982950 | 0.9736 |
Display the hyperparameters with the maximum test accuracy
[11]:
print("Best hyperparameters with %.2f%% test accuracy:" % (data_frame['accuracy_test'].max().values * 100,))
data_frame.loc[data_frame['accuracy_test'].idxmax()][vary.keys()]
Best hyperparameters with 97.36% test accuracy:
/var/folders/_0/dx4n8sh94h107t1d3g7wjwbm0000gn/T/ipykernel_49146/362537557.py:1: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.)
print("Best hyperparameters with %.2f%% test accuracy:" % (data_frame['accuracy_test'].max().values * 100,))
[11]:
| n_neurons | learning_rate | |
|---|---|---|
| 0 | 0 | |
| 5 | 128 | 0.015 |
[ ]: