The process of coaching neural networks is a difficult optimization process that can typically fail.

This will mean that the mannequin at the finish of the training cycle is probably not a secure or greatest performing set to make use of

One drawback to unravel this drawback is to make use of an common of a number of models seen in the direction of the end of the exercise. That is referred to as the Polyak-Ruppert imply and may be additional improved through the use of a linearly or exponentially reducing weighted common. Along with producing a extra secure model, the weighted energy of averaged fashions can even improve performance.

On this tutorial you’ll find out how we will mix the weights of several totally different fashions into one model for making predictions.

After completing this tutorial, you’ll know:

- The stochastic and challenging nature of neural community schooling might imply that the optimization course of shouldn’t be approaching.
- Creating a model where the common of detected patterns at the finish of the coaching session may result in a more secure and typically better-performing answer.
- The best way to develop ultimate fashions created with equal, linear, and exponentially weighted averages of model parameters of multiple stored models.
Contents

- 1 Overview of Guides
- 2 Average model weight mixture
- 3 Multi-grade score
- 4 Multi-Perceptron Model
- 5 Save Multiple Models to a File
- 6 New mannequin with common model weights
- 7 Predicting With an Average Mannequin Weight Ensemble
- 8 Linearly and Exponentially Reducing Weighted Average
- 9 Extensions
- 10 Additional Studying
- 11 Abstract
- 12 Develop Better Deep Studying Models At this time!

## Overview of Guides

This tutorial is divided into seven sections; they’re:

- Average unit weight
- Multi-grade score
- Multi-layer Perceptron mannequin
- Save a number of templates to file
- New model with average model weights
- Predicting common mannequin weight Ensemble
- Linear and exponentially reducing weighted common

## Average model weight mixture

Studying the weights of deep neural community fashions requires a decisive high-scale, non-convex optimization drawback.

The challenge of fixing this optimization is that there are various "good" solutions and that the learning algorithm can turn round and fail together. In the subject of stochastic optimization, this is known as issues in the approximation of the optimization algorithm to an answer in which the answer is decided by a set of sure weights.

Symptom that you may see in case you have an issue with mannequin convergence is the practice and / or check loss value that exhibits a bigger than anticipated deviation, eg

One solution to remedy this drawback is to combine the sizes collected at the end of the coaching. Usually, this can be termed a temporal average and known as a Polyak average or a Polyak-Ruppert average named for the unique developers of the technique.

Polyak averaging consists of averaging collectively a number of waypoint factors by way of the parameter state visited by the optimization algorithm. however it might prove to be a desirable answer particularly for very giant neural networks that may last for days, weeks and even months.

Essential progress was achieved on the foundation of a paradoxical concept: a sluggish algorithm with lower than optimum price of convergence have to be averaged.

– Average Acceleration of Stochastic Convergence, 1992.

The typical weight of multiple models in a single exercise visit impacts the peaceable impact of a loud optimization process, which can be noisy because of the selection of learning hyperparameters (e.g., Learning Velocity) or the type of a learning mapping perform. The result’s a remaining design or printing package deal that may provide a more secure and perhaps extra accurate end result.

The essential concept is that the optimization algorithm can bounce again and forth throughout the valley a number of occasions without going near the backside of the valley. The typical of all places on each side, nevertheless, must be near the bottom of the valley.

– Page 322, Deep Studying, 2016

The only implementation of the Polyak-Ruppert common consists of the average of the calculation. mannequin weights throughout final training durations

This can be improved by calculating a weighted average, the place extra weight is applied to the newest models which might be linearly lowered by means of earlier durations. An alternate and extra extensively used strategy is to make use of exponential decay at a weighted common

The polyak-Ruppert averaging has been shown to enhance normal SGD convergence […]. Alternatively, an exponential shifting average can be used in relation to the parameters, which provides a larger weight to a more moderen parameter worth

– Adam: a way for stochastic optimization, 2014.

Using a model weight average or weighted common The ultimate mannequin is in follow a basic method to ensure that coaching is obtained greatest outcomes. The strategy is one of the many "tricks" used in Google Inception V2 and V3 deep convolutional neural community models for photograph classification, a milestone in deep learning.

Model evaluations are carried out utilizing a calculated common.

– Reassessment of the Computing Structure, 2015.

### Would you like better results with deep studying?

Get a free 7-day e mail now (with sample code).

enroll and get a free PDF model of the E book course

Download a free mini-course

## Multi-grade score

We use a small multi-grade score drawback as the foundation

The Scikit-learning class provides a make_blobs () perform can be utilized to create a multi-class classification drawback with a specified quantity of samples, enter variables, courses, and variance of samples

The issue consists of two enter variables (representing the x and y coordinates of the points) and the normal deviation of 2.zero for each point. We use the similar randomness (seeds for a pseudorandom number generator) to ensure we all the time get the similar knowledge factors.

# creates second classification materials

X, y = make_blobs (n_samples = 1000, hubs = three, n_features = 2, cluster_std = 2, random_state = 2)# creates second classification

X, y = make_blobs (n_samples = 1000, hubs = 3 , n_features = 2, cluster_std = 2, random_state = 2)

The results are input and output parts of a database that we will model.

To get a sense of the complexity of the drawback, we will

The right instance is listed under

# scatter diagram of blob knowledge

from sklearn.datasets.samples_generator import make_blobs

from matplotlib import pyplot

import from number

# creates second classification info

X, y = make_blobs (n_samples = 1000, hubs = three, n_features = 2, cluster_std = 2, random_state = 2)

# scatter chart for each class value

class_value to (3):

# Choose level indexes with a category ID

row_ix = the place (y == class_value)

# scatter plot for factors with totally different colour

pyplot.scatter (X [row_ix, 0] X [row_ix, 1])

# Present plot

pyplot.present ()# scatter sample from blobs

from sklearn.datasets.samples_generator import make_blobs

matplotlib import pyplot

from numpy imports the place

# creates second classification material

X, y = make_blobs (n_samples = 1000, hubs = 3, n_features = 2, cluster_std = 2, random_state = 2)

# scatter sample for each class value

in category (three):

# select indexes of points with a class mark

row_ix = the place (y == class_value)

# scatter plot for totally different colours

pyplot.scatter (X [row_ix, 0] X [row_ix, 1])

# show plot

pyplot.show () [19659052] Executing an instance creates a scatter sample from the complete database. We will see that the normal deviation of 2.0 signifies that the courses aren’t linearly separable (separable on the line), causing many unclear points.

This is fascinating because it signifies that the drawback is non-trivial and allows the neural network model to seek out many various "good enough" candidate solutions that lead to giant variations

## Multi-Perceptron Model

Earlier than figuring out a mannequin, we have to clear up an issue that’s applicable for the group.

Our coaching material is comparatively small in our drawback. Particularly, there are 10: 1 relationship examples from the train database to holdout knowledge. This mimics a state of affairs where we will have a large number of unlabeled examples and a small number of labeled examples that can be utilized to coach the mannequin

. The mannequin is educated at the first 100 points, and the remaining 1000 is retained in a check database that isn’t out there in the model.

The problem is a multi-class score drawback, and our modeling with mushy activation is a perform on the output layer. Which means the model predicts a vector with three parts with the chance that the sample belongs to each of the three courses. That's why we’d like one scorching encode class values earlier than we divide the rows into the practice and check knowledge. We will do this using the Keras to_categorical () perform.

# creates second classification material

X, y = make_blobs (n_samples = 1100, centers = three, n_features = 2, cluster_std = 2, random_state = 2)

# one scorching coding supply variable

y = to_categorical (y)

# shared practice and check

n_train = 100

trainX, testX = X [:n_train, :] X [n_train:, :] trainy, testy = y [:n_train] y [n_train:]# creates second classification materials

X, y = make_blobs (n_samples = 1100, centers = 3, n_features = 2, cluster_std = 2, random_state = 2) [19659051] # one scorching code output variable

y = to_categorical (y)

# shared practice and check

n_train = 100

trainX, testX = X [:n_train, :] X [n_train:, :] 19659052] practice, testy = y [:n_train] y [n_train:]

Subsequent we will outline and assemble the model.

The model waits for samples with two enter variables. The mannequin then has one hidden layer with 25 knots and a corrected linear activation perform, then an output layer with three nodes, to predict the chance and delicate activation perform of each of the three courses.

As a result of the drawback is multi-class, we use a categorical loss of entropy to optimize the mannequin and the stochastic gradient descent at low learning velocity and velocity.

# outline mannequin

mannequin = Sequence ()

mannequin.add (dense (25, input_dim = 2, activation = & # 39; Relay & # 39;))

mannequin.add (Dense (3, activation = & # 39; softmax & # 39;))

choose = SGD (lr = zero.01, torque = 0.9)

mannequin.compile (loss = & # 39; categorical_crossentropy & # 39 ;, optimizer = choose, metrics = [‘accuracy’])# outline model

mannequin = consecutive ()

mannequin.add (density (25, input_dim = 2, activation = & # 39; Relay & # 39;))

mannequin.add (density (three, activation = & # 39; softmax & # 39;))

choose = SGD (lr = zero, 01, article = zero.9)

model.compile (loss = & # 39; category_crossentropy & # 39 ;, Optimizer = choose, metrics = [‘accuracy’])

The mannequin is suitable for 500 traineeships, and we consider the model for each check set utilizing a collection of exams as a validation collection.

# match mannequin

history = mannequin.fit (trainX, trainy, validation_data = (testX, testy), epochs = 500, verbose = 0)# fit mannequin

history = mannequin.match (trainX, trainy, validation_data = (testX, testy), epochs = 500, verbose = 0)

At the finish of the performance, we evaluated the performance of the mannequin on the practice and in the check collection.

# evaluate the model

_, train_acc = model.consider (trainX, trainee, verbose = 0)

_, test_acc = mannequin.evaluate (testX, testy, verbose = 0)

print (& # 39; Practice:% .3f, check:% .3f & # 39;% (train_acc, test_acc))# consider mannequin

_, train_acc = mannequin.consider (practice, practice, verbose = 0)

] _, test_acc = model.consider (testX, testy, verbose = zero)

print (Practice:% .3f, Check:% .3f & # 39;% (train_acc, test_acc ))

Lastly, the model's precision learning curves are plotted for every coaching session in both the coaching and validation knowledge sets.

pyplot.plot (historical past.historical past [‘acc’] label = & # 39; practice & # 39;)

pyplot.plot (historical past.historical past [‘val_acc’] label = & # 39; check & # 39;)

pyplot.legend ()

pyplot.show ()# learning curves with mannequin accuracy

pyplot.plot (history.historical past [‘acc’] label = & # 39; practice & # 39;)

pyplot.plot (history.history [‘val_acc’] sticker = & # 39; check & # 39;

pyplot.legend ()

pyplot.show ()

Combining all this collectively, the good instance is listed under.

# develops MLP for blob materials

from sklearn.datasets.samples_generator import make_blobs

from keras.utils import_categorical

from keras.fashions import Sequential

from keras.layers import Dense

from keras.optimizers At SGD

from matplotlib import pyplot

# creates second classification info

X, y = make_blobs (n_samples = 1100, facilities = three, n_features = 2, cluster_std = 2, random_state = 2)

# one scorching coding source variable

y = to_categorical (y)

# shared practice and check

n_train = 100

trainX, testX = X [:n_train, :] X [n_train:, :] trainee, testy = y [:n_train] y [n_train:] # Specify the template

model = Sequence ()

mannequin.add (dense (25, input_dim = 2, activation = & # 39; Relay & # 39;))

mannequin.add (Dense (three, activation = & # 39; softmax & # 39;))

choose = SGD (lr = 0.01, torque = 0.9)

mannequin.compile (loss = & # 39; category_crossentropy & # 39 ;, optimizer = choose, metrics = [‘accuracy’])

# fit mannequin

historical past = model.fit (trainX, training, validation_data = (testX, testy), episodes = 500, verbose = 0)

# evaluate the model

_, train_acc = model.evaluate (trainX, trainee, verbose = 0)

_, test_acc = model.evaluate (testX, testy, verbose = zero)

print (Practice:% .3f, check:% .3f & # 39;% (train_acc, test_acc))

# mannequin accuracy learning curves

pyplot.plot (historical past.history [‘acc’] label = & # 39; practice & # 39;)

pyplot.plot (historical past.historical past [‘val_acc’] label = & # 39; check & # 39;)

pyplot.legend ()

pyplot.show ()1

2

three

6

7

8

9

10

11 19

16

17

18

19

20

21

22

24

29

30

31

32

31

32

#

#

# was developed for ml of blob materials

from sklearn. datasets .samples_generator import make_blobs

from keras.utils import to_categorical

from keras.models import Sequential

from keras.layers import Density

from keras.optimizers import SGD

matplotlib import pyplot

# creates second classification materials

X, y = make_blobs (n_samples = 1100, hubs = 3, n_features = 2, cluster_std = 2, random_state = 2)

# one scorching code output variable

y = to_categorical (y )

# shared practice and check

n_train = 100

] trainX, testX = X [:n_train, :] X [n_train:, :]

training, testy = y [:n_train] y [n_train:]

# define model

model = consecutive ()

model.add (Dense (25, input_dim = 2, activation = & # 39; Relay & # 39;))

model.add (Dense (3, activation = & # 39; softmax & # 39;))

choose = SGD (lr = 0.01, torque = zero.9) [19659052] to mannequin.comp (loss = & # 39; categorical_crossentropy & # 39 ;, optimizer = choose , metrics = [‘accuracy’])

# matching model

history = mannequin.fit (trainX, trainy, validation_data = (testX, testy), episodes = 500, verbose = zero)

# price mannequin

_, train_acc = model.evaluate (trainX, trainy, verbose = zero)

_, test_acc = mannequin.evaluate (testX, testy, verbose = 0)

print (& # 39; Practice:% .3f, check:% .3f & # 39;% (train_acc, test_acc))

# mannequin precision learning curves

pyplot.plot (history.historical past [‘acc’] label = & # 39; practice & # 39;)

pyplot.plot (history.historical past [‘val_acc’] label = & # 39; check & # 39;)

pyplot.legend ()

pyplot. show ()

Executing an instance print output

Your outcomes differ (depending on the model!) given the high variation of the model

On this case, we will see that the mannequin reached about 86 Train Databases, which we all know are optimistic, % accuracy and about 81% of the check database, which we anticipate to be more lifelike

Practice: zero.860, Check: 0.812

Practice: 0.860, check: zero.812

There’s additionally a line graph displaying coaching curves with mannequin accuracy on the practice and check kits for every coaching session.

We will see that the accuracy of schooling is extra optimistic during most driving, as well as the last results. Importantly, during training, accuracy is detected to an inexpensive extent in both practice and check knowledge, which may present a superb foundation for calculating the average weight of the model.

## Save Multiple Models to a File

One Strategy to the Model Weight Unit is to Hold Models Weighted

There are three packages involved:

- It requires that you understand in advance learn how to combine the weights of the mannequin; perhaps you need to attempt totally different approaches.
- It requires you to know the number of eras used for training; perhaps you need to use an early cease
- It requires you to maintain at the very least one copy of the whole network in memory; This may be very expensive for giant fashions and fragile if the coaching course of crashes or kills.

An alternate is to save lots of the mannequin weights to the file throughout the exercise as the first step and later to mix the weights of the saved models to

Perhaps the easiest method to accomplish this is to manually apply the coaching process, one period at a time, then save the patterns at the finish of the interval if we’ve got exceeded the limit for the quantity of durations

For example, in our testing drawback, we practice the mannequin for 500 cycles and perhaps we will save patterns from interval 490 (eg between durations 490 and 499).

# fit model

n_epochs, n_save_after = 500, 490

for i in (n_epochs):

# match the pattern for one episode

mannequin.fit (trainX, training, episodes = 1, verbose = 0)

# Verify if the template must be saved

if i> = n_save_after:

model.save (model _ & # 39; + str (i) + & # 39; .h5 & # 39;)# matching model

n_epochs, n_save_after = 500, 490

for i region (n_epochs):

# fit mannequin for one episode

model.fit (trainX, trainy, epochs = 1, verbose = zero)

# verify in case you ought to save the model

if i> = n_save_after: [19659051] mannequin.save (& # 39; mannequin _ & # 39; + str (i) + & # 39; .h5 & # 39;)

Templates might be saved to a file using the template storage perform () and a filename containing the episode number .

Notice, Recording and Downloading Neural Network Templates to Keras requires that you’ve installed the h5py library. To put in this library with a pipette, comply with these steps:

By combining this all together, a perfect example of putting in a template in a coaching database and saving all the patterns from the last 10 episodes are listed under.

# save the templates to the end of the exercise

from sklearn.datasets.samples_generator import make_blobs

from keras.utils import_categorical

from keras.models import Sequential

from keras.layers import Dense

# creates second classification info

X, y = make_blobs (n_samples = 1100, centers = 3, n_features = 2, cluster_std = 2, random_state = 2)

# one scorching coding source variable

y = to_categorical (y)

# shared practice and check

n_train = 100

trainX, testX = X [:n_train, :] X [n_train:, :] trainee, testy = y [:n_train] y [n_train:] # Specify the template

model = Sequence ()

mannequin.add (dense (25, input_dim = 2, activation = & # 39; Relay & # 39;))

mannequin.add (Dense (three, activation = & # 39; softmax & # 39;))

model.compile (loss = & # 39; categorical_crossentropy & # 39 ;, optimizer = & # 39; metadata = [‘accuracy’])

# fit model

n_epochs, n_save_after = 500, 490

for i in (n_epochs):

# match the sample for one episode

model.fit (trainX, training, episodes = 1, verbose = 0)

# Examine if the template ought to be saved

if i> = n_save_after:

mannequin.save (model _ & # 39; str (i) + & # 39; h5 & # 39;)1

2

3

four 19659051] 7

9

10

11

12

13

14

15

16

17

22

23

24

25

26

#

#

#

# data designs for the end of the coaching course

sklearn.datasets.samples_generator import make_blobs

] from keras.utils import to_categorical

from keras.fashions import Sequential

from keras.layers import Dense

# creates second classification materials

X, y = make_blobs (n_samples = 1100, hubs = three, n_features = 2, cluster_std = 2, random_state = 2)

# one scorching code output variable

y = to_categorical (y)

# shared practice and check

n_train = 100

trainX , testX = X [:n_train, :] X [n_train:, :]

trainee, testy = y [:n_train] y [n_train:]

# outlined by mannequin

model = consecutive ()

mannequin.add (dense (25 , input_dim = 2, activation = & # 39; Relay & # 39;))

model.add (dense (three, activation = & # 39; softmax & # 39;)) [19659052] mannequin.comp (loss = & # 39; category_crossentropy & # 39 ;, optimizer = & # 39; adam & # 39; metrics = [‘accuracy’])

# matching mannequin

n_epochs, n_save_after = 500, 490

for i in the region (n_epochs) :

# adaptation model for one period

model.fit (trainX, trainy, epochs = 1, verbose = 0)

# examine if the model

ought to be saved if i> = n_save_after:

model. myself (& # 39; mannequin _ & # 39; + str (i) + & # 39; .h5 & # 39;)

Utilizing this instance saves 10 templates in your present work directory.

## New mannequin with common model weights

We will create a brand new mannequin from a number of present fashions with the similar structure.

First of all, we have to obtain the fashions. That is affordable because the fashions are small. When you work with very giant fashions, downloading fashions could be simpler than one by one and common weights in reminiscence.

Load_model () The Keras perform can be used to load a saved template from a file. The under load_all_models () perform downloads the templates from the present work listing. It takes the start and finish cycles as arguments so to check out totally different teams of patterns which are stored in adjoining sections.

# download mannequin from file

def load_all_models (n_start, n_end):

all_models = listing ()

section (n_start, n_end):

# specify the filename of this group

filename = & # 39; mannequin _ & # 39; + str (part) + & # 39; .h5 & # 39;

# uploads a template file

template = load_model (filename)

# add to membership record

all_models.append (template)

print (& # 39; uploaded% s & # 39;% filename)

restore all_models# obtain templates from file

def load_all_models (n_start, n_end):

all_models = record ()

to (n_start, n_end):

# specify filename for this ensemble

filename = & # 39; model _ & # 39; + str (part) + & # 39; .h5 & # 39;

# obtain mannequin from file

mannequin = load_model (filename)

# add to record

all_models. insertion (mannequin)

print (& # 39; loaded% s & # 39;% filename)

restore all_models

We will name the perform to obtain all fashions.

# download templates in order

member = load_all_models (490, 500)

print (& # 39; Downloaded% d templates %% (members))# download patterns in sequence

members = load_all_models (490, 500)

print (& # 39; Downloaded% d fashions% len (members ))

Once downloaded, we will create a brand new model with a weighted common of the model.

Each mannequin has a get_weights () perform that returns an inventory of tables, one for every template layer. We will record each layer of the mannequin, search the similar layer for every mannequin, and calculate the weighted common. This provides us a set of weights.

Then we will use the klone_model () Keras perform to create a clone of architecture and to play set_weights () averages to use. The model_weight_ensemble () under will do that.

# creates a template for the weight of multiple models

def model_weight_ensemble (members, weights):

# determines what number of layers are averaged

n_layers = len (members [0] .get_weights ())

# creates a set of average mannequin weights

avg_model_weights = listing ()

for layer (n_layers):

# Gather this layer for each model

layer_weights = array ([model.get_weights()[layer] in template members])

# weighted common of this layer weights

avg_layer_weights = common (layer_weights, axis = 0, weights = weights)

# Save common layer weights

avg_model_weights.append (avg_layer_weights)

# creates a brand new model with the similar structure

template = clone_model (members [0])

# Set the weights to the new one

mannequin.set_weights (avg_model_weights)

mannequin.compile (loss = & # 39; categorical_crossentropy & # 39 ;, optimizer = & # 39; metadata = [‘accuracy’])

return mannequin1

2

three

four

5

6

9

9

10

9

10

9 19

15

16

17

18

19

#

# creates a mannequin of the weight of a number of fashions

def model_weight_ensemble ( members, weights):

# determines how many layers are averaged

n_layers = len (members [0] .get_weights ())

# creates a set of common mannequin weights

avg_model_weights = listing () [19659052] to (n_layers):

# gather this layer for every mannequin

layer_weights = array ([model.get_weights()[layer] in model members])

# weighted common of this layer weight

avg_layer_weights = imply (layer_weights, axis = zero , weights = weights)

# save common flooring weights

avg_model_weights.append (avg_layer_w eights)

# creates a new mannequin with the similar construction [19659052] model = clone_model (members [0])

# sets weights in new

model.set_weights (avg_model_weights)

mannequin.comp (citadel = & # 39; categorical_crossentropy & # 39 ;, optimizer = & # 39; adam & # 39; metric ). The entire record is given under.

# average of the weight of multiple downloaded fashions

by keras.fashions import load_model

by keras.models import clone_model

common imports

from numpy import array# Obtain templates from file

def load_all_models (n_start, n_end):

all_models = record ()

section (n_start, n_end):

# specify the filename of this group

filename = & # 39; model _ & # 39; + str (section) + & # 39; .h5 & # 39;

# uploads a template file

template = load_model (filename)

# add to membership record

all_models.append (template)

print (& # 39; uploaded% s & # 39;% filename)

restore all_models# creates a template for the weight of a number of fashions

def model_weight_ensemble (members, weights):

# determines what number of layers are averaged

n_layers = len (members [0] .get_weights ())

# creates a set of common model weights

avg_model_weights = listing ()

for layer (n_layers):

# Acquire this layer for every mannequin

layer_weights = array ([model.get_weights()[layer] in template members])

# weighted average of this layer weights

avg_layer_weights = common (layer_weights, axis = zero, weights = weights)

# Save common layer weights

avg_model_weights.append (avg_layer_weights)

# creates a brand new mannequin with the similar construction

template = clone_model (members [0])

# Set the weights to the new one

mannequin.set_weights (avg_model_weights)

mannequin.compile (loss = & # 39; categorical_crossentropy & # 39 ;, optimizer = & # 39; metadata = [‘accuracy’])

recovery model# Download all templates

member = load_all_models (490, 500)

print (& # 39; Uploaded% d templates %% (members))

# Put together a set of equal weights

n_models = len (members)

weights = [1/n_models for i in range(1, n_models+1)] # creates a brand new mannequin with a weighted common of all model weights

model = model_weight_ensemble (jäsenet, painot)

# tiivistää luotu malli

mannequin.abstract ()1

2

3

four

5

6

7

eight

9

10

11 [19659052] 12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36 [19659052] 37

38

39

40

41

42

43

44

45

46

47

48

49

# keskiarvo useiden ladattujen mallien painoista

alkaen keras.models import load_model

alkaen keras.fashions import clone_model

numpy-tuonnin keskiarvosta

numpy-tuontiryhmästä

# latausmallit tiedostosta

def load_all_models (n_start, n_end):

all_models = lista ()

alueelle (n_start, n_end):

# define fi tälle yhtyeelle tarkoitettu leima

filename = 'malli_' + str (jakso) + '.h5'

# latausmalli tiedostosta

malli = load_model (tiedostonimi)

# lisää luetteloon jäseniä [19659052] all_models.append (malli)

print ('> ladattu% s'% tiedostonimi)

palauttaa all_models

# luo mallin useiden mallien painoista

def model_weight_ensemble (jäsenet, weights):

# decide what number of layers must be averaged

n_layers = len(members[0].get_weights())

# create an set of common model weights

avg_model_weights = listing()

for layer in range(n_layers):

# acquire this layer from each mannequin

layer_weights = array([model.get_weights()[layer] for mannequin in members])

# weighted average of weights for this layer[19659052] avg_layer_weights = common(layer_weights, axis=0, weights=weights)

# store common layer weights

avg_model_weights.append(avg_laye r_weights)

# create a new model with the similar construction

mannequin = clone_model(members[0])

# set the weights in the new

mannequin.set_weights(avg_model_weights)

model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=[‘accuracy’])

return model

# load all fashions into reminiscence

members = load_all_models(490, 500)

print('Loaded %d fashions' % len(members))

# prepare an array of equal weights

n_models = len(members)

weights = [1/n_models for i in range(1, n_models+1)]

# create a brand new mannequin with the weighted average of all model weights

model = model_weight_ensemble(members, weights)

# summarize the created model

mannequin.summary()

Operating the example first masses the 10 models from file.

>loaded model_490.h5

>loaded model_491.h5

>loaded model_492.h5

>loaded model_493.h5

>loaded model_494.h5

>loaded model_495.h5

>loaded model_496.h5

>loaded model_497.h5

>loaded model_498.h5

>loaded model_499.h5

Loaded 10 fashions>loaded model_490.h5

>loaded model_491.h5

>loaded model_492.h5

>loaded model_493.h5

>loaded model_494.h5

>loaded model_495.h5

>loaded model_496.h5

>loaded model_497.h5

>loaded model_498.h5

>loaded model_499.h5

Loaded 10 fashions

A model weight ensemble is created from these 10 models giving equal weight to each mannequin and a summary of the mannequin structure is reported.

_________________________________________________________________

Layer (sort) Output Form Param #

=================================================================

dense_1 (Dense) (None, 25) 75

_________________________________________________________________

dense_2 (Dense) (None, 3) 78

=================================================================

Complete params: 153

Trainable params: 153

Non-trainable params: 0

__________________________________________________________________________________________________________________________________

Layer (sort) Output Form Param #

=================================================================

dense_1 (Dense) (None, 25) 75

_________________________________________________________________

dense_2 (Dense) (None, 3) 78

=================================================================

Complete params: 153

Trainable params: 153

Non-trainable params: 0

_________________________________________________________________

## Predicting With an Average Mannequin Weight Ensemble

Now that we know how you can calculate a weighted average of model weights, we will evaluate predictions with the resulting model.

One concern is that we don’t know what number of fashions are applicable to mix in order to realize good performance. We will tackle this by evaluating mannequin weight averaging ensembles with the final n fashions and differ n to see how many fashions outcomes in good efficiency.

The evaluate_n_members() perform under will create a brand new mannequin from a given quantity of loaded models. Each mannequin is given an equal weight in contributing to the last mannequin, then the model_weight_ensemble() perform is known as to create the last model that’s then evaluated on the check dataset.

# evaluate a selected number of members in an ensemble

def evaluate_n_members(members, n_members, testX, testy):

# reverse loaded models so we build the ensemble with the final models first

members = record(reversed(members))

# select a subset of members

subset = members[:n_members] # prepare an array of equal weights

weights = [1.0/n_members for i in range(1, n_members+1)] # create a new model with the weighted common of all mannequin weights

model = model_weight_ensemble(subset, weights)

# make predictions and evaluate accuracy

_, test_acc = model.evaluate(testX, testy, verbose=zero)

return test_acc# evaluate a selected quantity of members in an ensemble

def evaluate_n_members(members, n_members, testX, testy):

# reverse loaded models so we construct the ensemble with the last fashions first

members = listing(reversed(members))

# choose a subset of members

subset = members[:n_members]

# put together an array of equal weights

weights = [1.0/n_members for i in range(1, n_members+1)]

# create a brand new model with the weighted common of all model weights

model = model_weight_ensemble(subset, weights)

# make predictions and evaluate accuracy

_, test_acc = mannequin.evaluate(testX, testy, verbose=0)

return test_acc

Importantly, the record of loaded models is reversed first to ensure that the last n models in the coaching run are used, which we might assume may need higher performance on common.

# reverse loaded models so we build the ensemble with the last models first

members = record(reversed(members))# reverse loaded models so we construct the ensemble with the last models first

members = listing(reversed(members))

We will then evaluate models created from totally different numbers of the final n models saved from the coaching run from the last 1-model to the last 10 models. Along with evaluating the combined ultimate model, we will also evaluate each saved standalone mannequin on the check dataset to match efficiency.

# consider totally different numbers of ensembles on hold out set

single_scores, ensemble_scores = listing(), record()

for i in vary(1, len(members)+1):

# evaluate mannequin with i members

ensemble_score = evaluate_n_members(members, i, testX, testy)

# evaluate the i'th mannequin standalone

_, single_score = members[i-1].evaluate(testX, testy, verbose=zero)

# summarize this step

print('> %d: single=%.3f, ensemble=%.3f' % (i, single_score, ensemble_score))

ensemble_scores.append(ensemble_score)

single_scores.append(single_score)# consider totally different numbers of ensembles on maintain out set

single_scores, ensemble_scores = record(), listing()

for i in range(1, len(members)+1):

# consider model with i members

ensemble_score = evaluate_n_members(members, i, testX, testy)

# consider the i'th model standalone

_, single_score = members[i-1].evaluate(testX, testy, verbose=0)

# summarize this step

print('> %d: single=%.3f, ensemble=%.3f' % (i, single_score, ensemble_score))

ensemble_scores.append(ensemble_score)

single_scores.append(single_score)

The collected scores may be plotted, with blue dots for the accuracy of the single saved models and the orange line for the check accuracy for the mannequin that combines the weights the final n models.

# plot score vs quantity of ensemble members

x_axis = [i for i in range(1, len(members)+1)] pyplot.plot(x_axis, single_scores, marker='o', linestyle='None')

pyplot.plot(x_axis, ensemble_scores, marker='o')

pyplot.present()# plot score vs quantity of ensemble members

x_axis = [i for i in range(1, len(members)+1)]

pyplot.plot(x_axis, single_scores, marker='o', linestyle='None')

pyplot.plot(x_axis, ensemble_scores, marker='o')

pyplot.show()

Tying all of this collectively, the complete example is listed under.

# common of mannequin weights on blobs drawback

from sklearn.datasets.samples_generator import make_blobs

from sklearn.metrics import accuracy_score

from keras.utils import to_categorical

from keras.models import load_model

from keras.fashions import clone_model

from keras.fashions import Sequential

from keras.layers import Dense

from matplotlib import pyplot

from numpy import common

from numpy import array# load models from file

def load_all_models(n_start, n_end):

all_models = listing()

for epoch in range(n_start, n_end):

# define filename for this ensemble

filename = 'model_' + str(epoch) + '.h5'

# load mannequin from file

model = load_model(filename)

# add to listing of members

all_models.append(model)

print('>loaded %s' % filename)

return all_models# # create a mannequin from the weights of multiple models

def model_weight_ensemble(members, weights):

# determine how many layers must be averaged

n_layers = len(members[0].get_weights())

# create an set of average model weights

avg_model_weights = listing()

for layer in vary(n_layers):

# gather this layer from each mannequin

layer_weights = array([model.get_weights()[layer] for mannequin in members])

# weighted common of weights for this layer

avg_layer_weights = common(layer_weights, axis=0, weights=weights)

# retailer average layer weights

avg_model_weights.append(avg_layer_weights)

# create a brand new mannequin with the similar construction

mannequin = clone_model(members[0])

# set the weights in the new

model.set_weights(avg_model_weights)

model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=[‘accuracy’])

return model# evaluate a selected number of members in an ensemble

def evaluate_n_members(members, n_members, testX, testy):

# choose a subset of members

subset = members[:n_members] # put together an array of equal weights

weights = [1.0/n_members for i in range(1, n_members+1)] # create a brand new mannequin with the weighted average of all mannequin weights

mannequin = model_weight_ensemble(subset, weights)

# make predictions and consider accuracy

_, test_acc = mannequin.evaluate(testX, testy, verbose=0)

return test_acc# generate second classification dataset

X, y = make_blobs(n_samples=1100, facilities=three, n_features=2, cluster_std=2, random_state=2)

# one scorching encode output variable

y = to_categorical(y)

# cut up into practice and check

n_train = 100

trainX, testX = X[:n_train, :]X[n_train:, :] trainy, testy = y[:n_train]y[n_train:] # load fashions in order

members = load_all_models(490, 500)

print('Loaded %d models' % len(members))

# reverse loaded models so we construct the ensemble with the last models first

members = record(reversed(members))

# consider totally different numbers of ensembles on hold out set

single_scores, ensemble_scores = record(), listing()

for i in vary(1, len(members)+1):

# consider model with i members

ensemble_score = evaluate_n_members(members, i, testX, testy)

# evaluate the i'th model standalone

_, single_score = members[i-1].consider(testX, testy, verbose=0)

# summarize this step

print('> %d: single=%.3f, ensemble=%.3f' % (i, single_score, ensemble_score))

ensemble_scores.append(ensemble_score)

single_scores.append(single_score)

# plot score vs quantity of ensemble members

x_axis = [i for i in range(1, len(members)+1)] pyplot.plot(x_axis, single_scores, marker='o', linestyle='None')

pyplot.plot(x_axis, ensemble_scores, marker='o')

pyplot.show()1

2

three

four

5

6

7

eight

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

# common of mannequin weights on blobs drawback

from sklearn.datasets.samples_generator import make_blobs

from sklearn.metrics import accuracy_score

from keras.utils import to_categorical

from keras.fashions import load_model

from keras.fashions import clone_model

from keras.fashions import Sequential

from keras.layers import Dense

from matplotlib import pyplot

from numpy import common

from numpy import array

# load models from file

def load_all_models(n_start, n_end):

all_models = record()

for epoch in vary(n_start, n_end):

# define filename for this ensemble

filename = 'model_' + str(epoch) + '.h5'

# load model from file

mannequin = load_model(filename)

# add to record of members

all_models.append(mannequin)

print('>loaded %s' % filename)

return all_models

# # create a mannequin from the weights of multiple fashions

def model_weight_ensemble(members, weights):

# decide how many layers have to be averaged

n_layers = len(members[0].get_weights())

# create an set of common mannequin weights

avg_model_weights = listing()

for layer in vary(n_layers):

# acquire this layer from every mannequin

layer_weights = array([model.get_weights()[layer] for mannequin in members])

# weighted average of weights for this layer

avg_layer_weights = average(layer_weights, axis=zero, weights=weights)

# retailer common layer weights

avg_model_weights.append(avg_layer_weights)

# create a new mannequin with the similar structure

model = clone_model(members[0])

# set the weights in the new

model.set_weights(avg_model_weights)

model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=[‘accuracy’])

return mannequin

# consider a selected quantity of members in an ensemble

def evaluate_n_members(members, n_members, testX, testy):

# choose a subset of members

subset = members[:n_members]

# prepare an array of equal weights

weights = [1.0/n_members for i in range(1, n_members+1)]

# create a new mannequin with the weighted common of all mannequin weights

model = model_weight_ensemble(subset, weights)

# make predictions and evaluate accuracy

_, test_acc = mannequin.evaluate(testX, testy, verbose=0)

return test_acc

# generate second classification dataset

X, y = make_blobs(n_samples=1100, facilities=3, n_features=2, cluster_std=2, random_state=2)

# one scorching encode output variable

y = to_categorical(y)

# cut up into practice and check

n_train = 100

trainX, testX = X[:n_train, :]X[n_train:, :]

trainy, testy = y[:n_train]y[n_train:]

# load fashions in order

members = load_all_models(490, 500)

print('Loaded %d fashions' % len(members))

# reverse loaded models so we construct the ensemble with the final fashions first

members = record(reversed(members))

# evaluate totally different numbers of ensembles on hold out set

single_scores, ensemble_scores = listing(), record()

for i in range(1, len(members)+1):

# evaluate mannequin with i members

ensemble_score = evaluate_n_members(members, i, testX, testy)

# consider the i'th mannequin standalone

_, single_score = members[i-1].evaluate(testX, testy, verbose=zero)

# summarize this step

print('> %d: single=%.3f, ensemble=%.3f' % (i, single_score, ensemble_score))

ensemble_scores.append(ensemble_score)

single_scores.append(single_score)

# plot score vs quantity of ensemble members

x_axis = [i for i in range(1, len(members)+1)]

pyplot.plot(x_axis, single_scores, marker='o', linestyle='None')

pyplot.plot(x_axis, ensemble_scores, marker='o')

pyplot.present()

Operating the instance first masses the 10 saved fashions.

>loaded model_490.h5

>loaded model_491.h5

>loaded model_492.h5

>loaded model_493.h5

>loaded model_494.h5

>loaded model_495.h5

>loaded model_496.h5

>loaded model_497.h5

>loaded model_498.h5

>loaded model_499.h5

Loaded 10 models>loaded model_490.h5

>loaded model_491.h5

>loaded model_492.h5

>loaded model_493.h5

>loaded model_494.h5

>loaded model_495.h5

>loaded model_496.h5

>loaded model_497.h5

>loaded model_498.h5

>loaded model_499.h5

Loaded 10 models

The performance of each individually saved mannequin is reported as well as an ensemble mannequin with weights averaged from all fashions up to and including every model, working backward from the finish of the coaching run.

The outcomes show that the greatest check accuracy was about 81.four% achieved by the final two models. We will see that the check accuracy of the model weight ensemble levels out the efficiency and performs simply as nicely.

Your specific outcomes will range based mostly on the fashions saved during the previous section.

> 1: single=zero.814, ensemble=0.814

> 2: single=0.814, ensemble=zero.814

> three: single=0.811, ensemble=0.813

> 4: single=0.805, ensemble=0.813

> 5: single=zero.807, ensemble=zero.811

> 6: single=0.805, ensemble=0.807

> 7: single=zero.802, ensemble=0.809

> eight: single=zero.805, ensemble=zero.808

> 9: single=0.805, ensemble=0.808

> 10: single=zero.810, ensemble=0.807> 1: single=0.814, ensemble=zero.814

> 2: single=0.814, ensemble=0.814

> 3: single=zero.811, ensemble=zero.813

> four: single=zero.805, ensemble=zero.813

> 5: single=0.807, ensemble=0.811

> 6: single=0.805, ensemble=zero.807

> 7: single=0.802, ensemble=zero.809

> 8: single=0.805, ensemble=zero.808

> 9: single=zero.805, ensemble=0.808

> 10: single=0.810, ensemble=zero.807

A line plot can also be created displaying the check accuracy of every single mannequin (blue dots) and the efficiency of the model weight ensemble (orange line).

We will see that averaging the mannequin weights does degree out the performance of the ultimate mannequin and performs at the very least as well as the remaining model of the run.

## Linearly and Exponentially Reducing Weighted Average

We will replace the instance and consider a linearly reducing weighting of the model weights in the ensemble.

The weights could be calculated as follows:

# prepare an array of linearly reducing weights

weights = [i/n_members for i in range(n_members, 0, -1)]# prepare an array of linearly reducing weights

weights = [i/n_members for i in range(n_members, 0, -1)]

This can be used as an alternative of the equal weights in the evaluate_n_members() perform.

The entire example is listed under.

# linearly reducing weighted average of models on blobs drawback

from sklearn.datasets.samples_generator import make_blobs

from sklearn.metrics import accuracy_score

from keras.utils import to_categorical

from keras.models import load_model

from keras.models import clone_model

from keras.models import Sequential

from keras.layers import Dense

from matplotlib import pyplot

from numpy import common

from numpy import array# load models from file

def load_all_models(n_start, n_end):

all_models = listing()

for epoch in vary(n_start, n_end):

# define filename for this ensemble

filename = 'model_' + str(epoch) + '.h5'

# load model from file

mannequin = load_model(filename)

# add to listing of members

all_models.append(model)

print('>loaded %s' % filename)

return all_models# create a mannequin from the weights of a number of models

def model_weight_ensemble(members, weights):

# decide how many layers must be averaged

n_layers = len(members[0].get_weights())

# create an set of average mannequin weights

avg_model_weights = record()

for layer in vary(n_layers):

# acquire this layer from each mannequin

layer_weights = array([model.get_weights()[layer] for mannequin in members])

# weighted average of weights for this layer

avg_layer_weights = average(layer_weights, axis=0, weights=weights)

# retailer common layer weights

avg_model_weights.append(avg_layer_weights)

# create a new mannequin with the similar construction

model = clone_model(members[0])

# set the weights in the new

mannequin.set_weights(avg_model_weights)

mannequin.compile(loss='categorical_crossentropy', optimizer='adam', metrics=[‘accuracy’])

return model# evaluate a selected number of members in an ensemble

def evaluate_n_members(members, n_members, testX, testy):

# select a subset of members

subset = members[:n_members] # prepare an array of linearly reducing weights

weights = [i/n_members for i in range(n_members, 0, -1)] # create a new model with the weighted common of all model weights

model = model_weight_ensemble(subset, weights)

# make predictions and consider accuracy

_, test_acc = model.evaluate(testX, testy, verbose=0)

return test_acc# generate second classification dataset

X, y = make_blobs(n_samples=1100, facilities=3, n_features=2, cluster_std=2, random_state=2)

# one scorching encode output variable

y = to_categorical(y)

# cut up into practice and check

n_train = 100

trainX, testX = X[:n_train, :]X[n_train:, :] trainy, testy = y[:n_train]y[n_train:] # load models in order

members = load_all_models(490, 500)

print('Loaded %d fashions' % len(members))

# reverse loaded models so we construct the ensemble with the last models first

members = record(reversed(members))

# consider totally different numbers of ensembles on hold out set

single_scores, ensemble_scores = record(), listing()

for i in vary(1, len(members)+1):

# consider mannequin with i members

ensemble_score = evaluate_n_members(members, i, testX, testy)

# evaluate the i'th model standalone

_, single_score = members[i-1].consider(testX, testy, verbose=0)

# summarize this step

print('> %d: single=%.3f, ensemble=%.3f' % (i, single_score, ensemble_score))

ensemble_scores.append(ensemble_score)

single_scores.append(single_score)

# plot rating vs number of ensemble members

x_axis = [i for i in range(1, len(members)+1)] pyplot.plot(x_axis, single_scores, marker='o', linestyle='None')

pyplot.plot(x_axis, ensemble_scores, marker='o')

pyplot.present()1

2

three

four

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

# linearly reducing weighted common of models on blobs drawback

from sklearn.datasets.samples_generator import make_blobs

from sklearn.metrics import accuracy_score

from keras.utils import to_categorical

from keras.models import load_model

from keras.fashions import clone_model

from keras.models import Sequential

from keras.layers import Dense

from matplotlib import pyplot

from numpy import average

from numpy import array

# load models from file

def load_all_models(n_start, n_end):

all_models = listing()

for epoch in vary(n_start, n_end):

# define filename for this ensemble

filename = 'model_' + str(epoch) + '.h5'

# load model from file

mannequin = load_model(filename)

# add to record of members

all_models.append(model)

print('>loaded %s' % filename)

return all_models

# create a mannequin from the weights of multiple fashions

def model_weight_ensemble(members, weights):

# determine what number of layers have to be averaged

n_layers = len(members[0].get_weights())

# create an set of common model weights

avg_model_weights = listing()

for layer in range(n_layers):

# acquire this layer from every model

layer_weights = array([model.get_weights()[layer] for mannequin in members])

# weighted common of weights for this layer

avg_layer_weights = average(layer_weights, axis=zero, weights=weights)

# store average layer weights

avg_model_weights.append(avg_layer_weights)

# create a new model with the similar structure

mannequin = clone_model(members[0])

# set the weights in the new

model.set_weights(avg_model_weights)

model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=[‘accuracy’])

return mannequin

# evaluate a selected number of members in an ensemble

def evaluate_n_members(members, n_members, testX, testy):

# choose a subset of members

subset = members[:n_members]

# put together an array of linearly reducing weights

weights = [i/n_members for i in range(n_members, 0, -1)]

# create a brand new mannequin with the weighted common of all mannequin weights

mannequin = model_weight_ensemble(subset, weights)

# make predictions and consider accuracy

_, test_acc = mannequin.consider(testX, testy, verbose=zero)

return test_acc

# generate second classification dataset

X, y = make_blobs(n_samples=1100, centers=three, n_features=2, cluster_std=2, random_state=2)

# one scorching encode output variable

y = to_categorical(y)

# cut up into practice and check

n_train = 100

trainX, testX = X[:n_train, :]X[n_train:, :]

trainy, testy = y[:n_train]y[n_train:]

# load fashions in order

members = load_all_models(490, 500)

print('Loaded %d fashions' % len(members))

# reverse loaded fashions so we build the ensemble with the final fashions first

members = record(reversed(members))

# consider totally different numbers of ensembles on hold out set

single_scores, ensemble_scores = listing(), record()

for i in vary(1, len(members)+1):

# evaluate mannequin with i members

ensemble_score = evaluate_n_members(members, i, testX, testy)

# consider the i'th model standalone

_, single_score = members[i-1].evaluate(testX, testy, verbose=zero)

# summarize this step

print('> %d: single=%.3f, ensemble=%.3f' % (i, single_score, ensemble_score))

ensemble_scores.append(ensemble_score)

single_scores.append(single_score)

# plot rating vs number of ensemble members

x_axis = [i for i in range(1, len(members)+1)]

pyplot.plot(x_axis, single_scores, marker='o', linestyle='None')

pyplot.plot(x_axis, ensemble_scores, marker='o')

pyplot.show()

Operating the example stories the efficiency of every single model once more, and this time the check accuracy of each average mannequin weight ensemble with a linearly reducing contribution of models.

We will see that, at the very least in this case, the ensemble achieves a small bump in efficiency above any standalone model to about 81.5% accuracy.

…

> 1: single=zero.814, ensemble=zero.814

> 2: single=zero.814, ensemble=0.815

> 3: single=zero.811, ensemble=0.814

> four: single=zero.805, ensemble=0.813

> 5: single=0.807, ensemble=0.813

> 6: single=zero.805, ensemble=zero.813

> 7: single=zero.802, ensemble=zero.811

> 8: single=0.805, ensemble=zero.810

> 9: single=0.805, ensemble=0.809

> 10: single=zero.810, ensemble=0.809…

> 1: single=0.814, ensemble=0.814

> 2: single=zero.814, ensemble=0.815

> 3: single=0.811, ensemble=0.814

> 4: single=zero.805, ensemble=0.813

> 5: single=zero.807, ensemble=0.813

> 6: single=zero.805, ensemble=0.813

> 7: single=0.802, ensemble=zero.811

> eight: single=zero.805, ensemble=zero.810

> 9: single=0.805, ensemble=zero.809

> 10: single=0.810, ensemble=0.809

The road plot exhibits the bump in efficiency and exhibits a more secure performance in terms of check accuracy over the totally different sized ensembles created, as compared to the use of an evenly weighted ensemble.

We will also experiment with an exponential decay of the contribution of models. This requires that a decay fee (alpha) is specified. The example under creates weights for an exponential decay with a decrease price of 2.

# put together an array of exponentially reducing weights

alpha = 2.zero

weights = [exp(-i/alpha) for i in range(1, n_members+1)]# prepare an array of exponentially reducing weights

alpha = 2.0

weights = [exp(-i/alpha) for i in range(1, n_members+1)]

The entire instance with an exponential decay for the contribution of models to the average weights in the ensemble model is listed under.

# exponentially reducing weighted average of fashions on blobs drawback

from sklearn.datasets.samples_generator import make_blobs

from sklearn.metrics import accuracy_score

from keras.utils import to_categorical

from keras.models import load_model

from keras.models import clone_model

from keras.fashions import Sequential

from keras.layers import Dense

from matplotlib import pyplot

from numpy import average

from numpy import array

from math import exp# load models from file

def load_all_models(n_start, n_end):

all_models = record()

for epoch in range(n_start, n_end):

# define filename for this ensemble

filename = 'model_' + str(epoch) + '.h5'

# load mannequin from file

model = load_model(filename)

# add to listing of members

all_models.append(mannequin)

print('>loaded %s' % filename)

return all_models# create a mannequin from the weights of a number of models

def model_weight_ensemble(members, weights):

# decide how many layers have to be averaged

n_layers = len(members[0].get_weights())

# create an set of common mannequin weights

avg_model_weights = listing()

for layer in vary(n_layers):

# gather this layer from every mannequin

layer_weights = array([model.get_weights()[layer] for model in members])

# weighted common of weights for this layer

avg_layer_weights = average(layer_weights, axis=0, weights=weights)

# store average layer weights

avg_model_weights.append(avg_layer_weights)

# create a new model with the similar construction

mannequin = clone_model(members[0])

# set the weights in the new

model.set_weights(avg_model_weights)

mannequin.compile(loss='categorical_crossentropy', optimizer='adam', metrics=[‘accuracy’])

return mannequin# consider a selected quantity of members in an ensemble

def evaluate_n_members(members, n_members, testX, testy):

# choose a subset of members

subset = members[:n_members] # put together an array of exponentially reducing weights

alpha = 2.zero

weights = [exp(-i/alpha) for i in range(1, n_members+1)] # create a new mannequin with the weighted common of all mannequin weights

mannequin = model_weight_ensemble(subset, weights)

# make predictions and consider accuracy

_, test_acc = mannequin.evaluate(testX, testy, verbose=zero)

return test_acc# generate second classification dataset

X, y = make_blobs(n_samples=1100, centers=3, n_features=2, cluster_std=2, random_state=2)

# one scorching encode output variable

y = to_categorical(y)

# cut up into practice and check

n_train = 100

trainX, testX = X[:n_train, :]X[n_train:, :] trainy, testy = y[:n_train]y[n_train:] # load models in order

members = load_all_models(490, 500)

print('Loaded %d fashions' % len(members))

# reverse loaded fashions so we build the ensemble with the final fashions first

members = listing(reversed(members))

# evaluate totally different numbers of ensembles on maintain out set

single_scores, ensemble_scores = record(), record()

for i in range(1, len(members)+1):

# evaluate model with i members

ensemble_score = evaluate_n_members(members, i, testX, testy)

# evaluate the i'th model standalone

_, single_score = members[i-1].consider(testX, testy, verbose=zero)

# summarize this step

print('> %d: single=%.3f, ensemble=%.3f' % (i, single_score, ensemble_score))

ensemble_scores.append(ensemble_score)

single_scores.append(single_score)

# plot score vs number of ensemble members

x_axis = [i for i in range(1, len(members)+1)] pyplot.plot(x_axis, single_scores, marker='o', linestyle='None')

pyplot.plot(x_axis, ensemble_scores, marker='o')

pyplot.show()1

2

three

four

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

# exponentially reducing weighted common of models on blobs drawback

from sklearn.datasets.samples_generator import make_blobs

from sklearn.metrics import accuracy_score

from keras.utils import to_categorical

from keras.fashions import load_model

from keras.fashions import clone_model

from keras.models import Sequential

from keras.layers import Dense

from matplotlib import pyplot

from numpy import common

from numpy import array

from math import exp

# load fashions from file

def load_all_models(n_start, n_end):

all_models = record()

for epoch in range(n_start, n_end):

# outline filename for this ensemble

filename = 'model_' + str(epoch) + '.h5'

# load mannequin from file

mannequin = load_model(filename)

# add to record of members

all_models.append(model)

print('>loaded %s' % filename)

return all_models

# create a mannequin from the weights of a number of models

def model_weight_ensemble(members, weights):

# decide how many layers have to be averaged

n_layers = len(members[0].get_weights())

# create an set of average mannequin weights

avg_model_weights = listing()

for layer in vary(n_layers):

# acquire this layer from each mannequin

layer_weights = array([model.get_weights()[layer] for mannequin in members])

# weighted average of weights for this layer

avg_layer_weights = average(layer_weights, axis=zero, weights=weights)

# store common layer weights

avg_model_weights.append(avg_layer_weights)

# create a brand new model with the similar structure

model = clone_model(members[0])

# set the weights in the new

mannequin.set_weights(avg_model_weights)

mannequin.compile(loss='categorical_crossentropy', optimizer='adam', metrics=[‘accuracy’])

return mannequin

# consider a selected number of members in an ensemble

def evaluate_n_members(members, n_members, testX, testy):

# choose a subset of members

subset = members[:n_members]

# put together an array of exponentially reducing weights

alpha = 2.zero

weights = [exp(-i/alpha) for i in range(1, n_members+1)]

# create a brand new model with the weighted average of all mannequin weights

model = model_weight_ensemble(subset, weights)

# make predictions and evaluate accuracy

_, test_acc = model.evaluate(testX, testy, verbose=0)

return test_acc

# generate second classification dataset

X, y = make_blobs(n_samples=1100, facilities=3, n_features=2, cluster_std=2, random_state=2)

# one scorching encode output variable

y = to_categorical(y)

# cut up into practice and check

n_train = 100

trainX, testX = X[:n_train, :]X[n_train:, :]

trainy, testy = y[:n_train]y[n_train:]

# load models in order

members = load_all_models(490, 500)

print('Loaded %d fashions' % len(members))

# reverse loaded models so we build the ensemble with the last models first

members = record(reversed(members))

# evaluate totally different numbers of ensembles on hold out set

single_scores, ensemble_scores = record(), listing()

for i in range(1, len(members)+1):

# evaluate model with i members

ensemble_score = evaluate_n_members(members, i, testX, testy)

# evaluate the i'th model standalone

_, single_score = members[i-1].evaluate(testX, testy, verbose=0)

# summarize this step

print('> %d: single=%.3f, ensemble=%.3f' % (i, single_score, ensemble_score))

ensemble_scores.append(ensemble_score)

single_scores.append(single_score)

# plot score vs number of ensemble members

x_axis = [i for i in range(1, len(members)+1)]

pyplot.plot(x_axis, single_scores, marker='o', linestyle='None')

pyplot.plot(x_axis, ensemble_scores, marker='o')

pyplot.present()

Operating the instance exhibits a small enchancment in performance very similar to the use of a linear decay in the weighted average of the saved fashions.

> 1: single=0.814, ensemble=0.814

> 2: single=zero.814, ensemble=0.815

> 3: single=0.811, ensemble=zero.814

> 4: single=zero.805, ensemble=0.814

> 5: single=0.807, ensemble=zero.813

> 6: single=zero.805, ensemble=0.813

> 7: single=0.802, ensemble=zero.813

> 8: single=zero.805, ensemble=0.813

> 9: single=zero.805, ensemble=0.813

> 10: single=0.810, ensemble=0.813> 1: single=0.814, ensemble=zero.814

> 2: single=zero.814, ensemble=zero.815

> 3: single=0.811, ensemble=0.814

> four: single=zero.805, ensemble=0.814

> 5: single=zero.807, ensemble=zero.813

> 6: single=0.805, ensemble=0.813

> 7: single=zero.802, ensemble=0.813

> eight: single=zero.805, ensemble=0.813

> 9: single=0.805, ensemble=zero.813

> 10: single=0.810, ensemble=0.813

The road plot of the check accuracy scores exhibits the stronger stabilizing effect of using the exponential decay as an alternative of the linear or equal weighting of models.

## Extensions

This part lists some ideas for extending the tutorial that you may wish to explore.

- Quantity of Models. Evaluate the effect of many more models contributing their weights to the last model.
- Decay Fee. Evaluate the effect on check performance of using totally different decay charges for an exponentially weighted common.

Should you discover any of these extensions, I’d love to know.

## Additional Studying

This part offers more assets on the matter in case you are trying to go deeper.

### Books

### Papers

### API

### Articles

## Abstract

In this tutorial, you found the way to combine the weights from a number of totally different fashions into a single mannequin for making predictions.

Specifically, you discovered:

- The stochastic and difficult nature of coaching neural networks can imply that the optimization process doesn’t converge.
- Creating a model with the average of the weights from fashions noticed in the direction of the end of a training run may result in a more secure and typically better-performing answer.
- Find out how to develop remaining models created with the equal, linearly, and exponentially weighted average of model parameters from multiple saved models.

Do you’ve any questions?

Ask your questions in the feedback under and I’ll do my greatest to reply.