Добавил:
ivanov666
Опубликованный материал нарушает ваши авторские права? Сообщите нам.
Вуз:
Предмет:
Файл:Intelligent data analysis in medicine. Study aid
.pdf
31
CHAPTER 4. MACHINE LEARNING ALGORITHMS
Statement of the machine learning problem
The function that the model learns is the main task of machine learning. This
does not refer to the task of actual learning, but rather to the task at hand. For example,
a robot vacuum cleaner will have the task of vacuuming a surface. Often a task is
broken down into smaller subtasks.
Facilities and libraries, implementing machine learning algorithms
Python is becoming an increasingly popular tool in the fields of data science and
machine learning. All libraries are open source on Github (the largest web service for
hosting IT projects and their joint development), which also provides indicators of
popularity and reliability. Here are the main Python libraries used for data mining:
1. Numerical Python, or NumPy, is an important Python extension library. It
provides fast manipulation of N-dimensional array objects for vector arithmetic,
mathematical functions, linear algebra, and random number generation. Basic
arithmetic performance for processing large data sets is significantly lower without this
library.
2. SciPy is a scientific data processing library based on NumPy. It includes linear
algebra procedures, as well as signal and image processing, ordinary differential
equation methods, and special functions.
3. Matplotlib is a Python library with implemented methods for plotting and data
visualization. At the same time, the simplicity and usefulness of Matplotlib help Python
become an alternative to Scilab.
4. Pandas is a tool for data aggregation, data processing and data visualization.
5. Scikit-learn contains image processing and machine learning methods. This
library is built on SciPy and includes clustering, classification and regression
algorithms. This includes many of the algorithms discussed in this chapter, such as
Naive Bayes, decision trees, random forests, k-means, and support vector machines.
6. TensorFlow is an open-source, Google-supported data mining library
optimized for machine learning tasks. This allows the use of multi-layer neural
networks and deep learning algorithms. TensorFlow is used in many Google intelligent
32
platforms. TensorFlow also allows the use of parallel computing technologies on
Nvidia graphics cards.
7. Keras is a library for building neural networks based on TensorFlow.
Supervised learning algorithms
In many settings, the main goal of supervised learning is to fine-tune the
predictor function f(x) or derive a hypothesis. Training involves the use of
mathematical algorithms to represent input data x in a specific domain. For example, x
could be the time of day, and f(x) could be a prediction of the waiting time at a particular
hospital. Most often in practice, x represents multiple data points. Therefore, the input
data is expressed as a vector. For example, in the previous example, f(x) takes as input
x as the day of the week. By improving the predictor, we could use not only the day
(x1), but also time (x2), weather (x3), place in queue (x4), etc. provided that the data is
available.
Deciding which inputs to use is an integral part of the machine learning
information system design process.
Each i-th record in the data set is a vector of features x(i). In the case of using
the supervised learning method, each instance will be associated with a target label y(i).
The model is trained with input data in the form (x(i), y(i)). The training data set can
be represented as {(x(i), y(i)); i = 1, ..., N}, where x is the input and y is the output.
The model has a simple functional form: f(x) = ax + b, where a and b are
constants related to the noise in the original data. Machine learning aims to find the
optimal values of a and b so that the predictions of f(x) are as accurate as possible. This
teaching method is also called inductive. The model iterates through the instances in
the training data set to identify different kinds of patterns in them. It derives a pattern
or hypothesis from a set of examples. The output of a machine learning model for
prediction problems differs from the actual value of the function due to bias, noise, and
variance. Optimization of f(x) is carried out using a training set of examples. Each
training example {x1, ..., xn} has an input value x
train
and a corresponding output value
Y. Each example is processed and the difference between the known and correct value
y (where y ∈ Y) and the predicted value f(h
train
) is calculated. After processing the
required number of samples from the training data set, the error rate f(x) is calculated,
and the values of a and b are used as factors to improve the accuracy of the prediction.
33
This also reduces the inherent randomness and/or noise in the data, also known as fatal
errors. There are two main approaches to feature selection:
– independent assessment based on general characteristics of the data. Methods
related to this approach are called filtering methods because the set of features is
filtered before building the model;
– using a machine learning algorithm to evaluate different subsets of features
and select the one with the best classification accuracy. The latter algorithm will
ultimately be used to build a predictive model. Methods in this category are called
wrapper methods because the resulting algorithm wraps the entire feature selection
process.
To build forecasts based on machine learning methods, the theorem on the law
of large numbers is used. The theorem describes how the results of a large number of
identical experiments eventually converge. This process is repeated until the system
converges to the best values for a and b. Thus, the system learns from its own
experience and prepares for further deployment. Typically, we are interested in
minimizing model error or making predictions as accurate as possible. There are two
main approaches in supervised machine learning: regression-based systems and
classification-based systems.
Classification
Classification is the process of determiningation of the class of an element
(or elements — x) from a set of discrete predefined categories (or labels — y) by
approximating a mapping function — f. Classification algorithms use features (or
attributes) to determine how to classify an element into at least two or more classes
(Fig. 6).
The training data is provided as an input vector. It consists of labels that are
characteristics of the element. For example, the element might read like this: {x
train
,
output}. It is on the basis of the data set that the algorithms are trained to classify. Most
classification algorithms require examples to be labeled to identify patterns; therefore,
the classification accuracy of the model is best verified by correctly classified examples
of all predictions made. Examples of algorithms that perform classification are decision
trees, naive Bayes classifier, logistic regression, kNN (k-nearest neighbors), support
vector machine, etc.
34
Fig. 6. Graphical interpretation of classification and regression problems
Regression
Regression involves prediction of an outcome, which is a continuous variable.
Since the regression-based system predicts the value, performance is measured by
assessing the number of errors in the made predictions. Coefficient of determination,
mean absolute error, relative absolute error, and root mean square error are statistical
metrics used to evaluate regression models.
Algorithms that can perform regression include linear regression, decision trees,
support vector machines, k-nearest neighbors, and perceptrons. It is often possible to
convert problems between classification and regression. For example, some patient
characteristic may also be useful if it is stored as a continuous value. A continuous
quantity can be classified into a discrete category according to certain rules. Often time
series refers to input data that is provided as a sequence of discrete values. The
sequential order of the data adds an additional dimension of information. Time series
regression problems are standard in data analysis theory.
Decision trees
Decision trees are diagrams that demonstrate the decision-making process in the
form of rules for categorization performing. Decision trees start at the root and contain
internal nodes representing functions and branches representing results. Thus, decision
trees allow solving classification problems. Decision trees can be used to facilitate
understanding of classification mechanisms.
35
Each decision tree is a disjunction of implications (i.e., if-then statements), and
the implications are Horn sentences useful for logic programming. Horn's clause is a
disjunction of literals. Assuming there are no errors in the data in the form of
inconsistencies, we can always build a decision tree to train data sets with 100 %
accuracy. However, this may not work in the real world and may indicate errors in the
original sample, overfitting, and other problems.
The main issue in training a decision tree is which nodes should be placed in
which positions, including the root node and decision nodes. There are three main
algorithms for constructing a decision tree. The difference between the algorithms lies
in the choice of measure or cost function for which nodes or functions are defined. The
root is the top node. The tree is divided into branches that are evaluated through a cost
function, and the branch that is not split is the leaf node, decision, or leaf.
Decision trees are useful because the knowledge gained can be expressed in an
easy-to-read and understand format. They simulate human decision making, with clear
priority determined by the importance of functions, relationships, and decisions. They
are simple in the sense that the results can be expressed as a set of rules. Decision trees
provide an advantage in the way they can represent large data sets and prioritize the
most relevant features. If the depth of the decision tree is not specified, it will
eventually examine the given data and fit it. It is recommended to set a small depth of
decision tree modeling. Alternatively, the decision tree can be pruned, usually starting
with the least important feature or by incorporating dimensionality reduction
techniques.
Overfitting is a classic problem in machine learning, and it's not unique to
decision trees. All algorithms are at risk of overfitting, and there are many techniques
to solve this problem. Random forest based algorithms can be extremely useful for this.
Space dimensionality reduction reduces the size of the decision tree by removing
features that provide the least information. As a result, the classification rules become
less complex and improve prediction accuracy. Model accuracy is calculated as the
percentage of correctly classified examples for the test dataset.
Random forest
Python: scikit-learn;
Method: RandomForestClassifier.
36
A random forest is created by training multiple decision trees simultaneously.
The main purpose of using random forest is to prevent overfitting. The more decision
trees there are in a random forest, the more accurate the results. Each random forest
takes a sample of the data set and a random subset of features to make a decision.
Random forest algorithms can be used for classification and regression problems. In
regression problems, the average of the results is taken, while in classification
problems, the majority category is taken. A random forest decision tree is an ensemble
of different models. The use of the ensemble method reduces the variance in the
algorithm by combining forecasts from several models (Fig. 7).
Fig. 7. Graphical interpretation of methods based on decision trees
The random forest model works especially well when the input data is limited
by classification. The advantage of random forest classifiers is that they can also handle
data samples containing missing values.
In addition, the choice of algorithm stopping criterion is important. The learning
algorithm can work until it obtains “pure” subsets with examples of the same class. In
this case, there is a high probability of getting a tree in which a separate leaf will be
created for each example. Such a tree cannot be used in practice due to overfitting.
Each example will have its own unique path in the tree. The result is a set of rules that
is relevant only for this example.
Overfitting in the case of a decision tree has consequences similar to those of
neural networks. It will accurately recognize examples from training, but will not be
37
able to work with new data. Another disadvantage is that the structure of the retrained
tree is complex and difficult to interpret. The developers forcefully stop building the
tree to prevent it from becoming “overfitted.”
Several approaches are used for this:
– early stop. The algorithm stops after reaching a specified criterion value (for
example, the percentage of correctly recognized examples). The advantage of the
method is the reduction of time spent on training. The main disadvantage is that
stopping early has a negative impact on the accuracy of the tree. Because of this, many
experts advise giving preference to the branch cutting method;
– limitation of tree depth. The algorithm stops after reaching the specified
number of splits in branches. This approach also has a negative impact on the tree's
accuracy;
– setting the minimum allowed number of examples in a node. A restriction is
set on creating nodes with less than a specified number of examples. In this case, trivial
partitions and insignificant rules will not be created.
These approaches are rarely used because they do not guarantee a good enough
result. Most often they only work in some special cases. There are no guidelines for
using any method, so analysts must gain practical experience through trial and error.
Advantages of decision trees usage:
– formulate clear and understandable classification rules. That is, decision trees
are well and quickly interpreted;
– are able to generate rules in areas where it is difficult for a specialist to
formalize his knowledge;
– are easily visualized, i.e. can be “interpreted” not only as a model as a whole,
but also as a prediction for an individual test subject (path in the tree);
– learn quickly and make forecasts;
– many model parameters are not required;
– support both numeric and categorical features.
Disadvantages of using decision trees:
– decision trees are sensitive to noise in the input data. Small changes in the
training sample can lead to global adjustments of the model, which will affect the
change in classification rules and interpretability of the model;
– the dividing boundary has certain limitations, which is why the decision tree
is inferior to other methods in terms of classification quality;
38
– it is possible to retrain the decision tree, which is why you have to resort to the
method of cutting off branches, setting the minimum number of elements in the leaves
of the tree or the maximum depth of the tree;
– complex search for an optimal decision tree. This leads to the need to use
heuristics such as greedy search for a feature with maximum information gain, which
ultimately does not provide the maximum guarantee of finding the optimal tree;
– the decision tree gives a constant forecast for objects located in the feature
space outside the parallelepiped, which does not cover all objects of the training
sample.
Boosting
Boosting is an ensemble method that iteratively builds a set of predictive models.
The idea of this method assumes that each new model learns from the mistakes of
previous models. Boosting is usually used when there is a large sample of data to
predict. Each iteration of the method identifies relationships in the data, and the result
is then analyzed for misclassification errors, which are assigned specific weights. The
goal is to minimize the error of the previous model. The underlying machine learning
method used to improve the quality of algorithms can be anything.
Improving the quality of algorithms is performed in the following order:
1. The original data is processed.
2. The decision on classification is made based on the most significant cross-
section of characteristics.
3. Weights are assigned to incorrectly classified observations.
4. The process is repeated and all iterations are combined to obtain the final
classifier.
Main advantages of boosting:
– simplicity of software implementation of the algorithm;
– boosting has algorithms that are easy to understand and interpret and can learn
from their mistakes. These algorithms require no data preprocessing and also have
built-in procedures for handling missing values. In addition, most languages have builtin libraries for implementing boosting algorithms with many parameters that allow you
to precisely set performance;
39
– reduction of displacement. Bias is the presence of uncertainty or imprecision
in machine learning results. Boosting algorithms combine multiple weak models into
a sequential method that iteratively improves observations. This approach helps reduce
the high bias that is common in machine learning models;
– computational efficiency of the algorithm. Boosting algorithms focus on
elements that improve prediction accuracy during training. They are capable of
reducing the number of data attributes and efficiently processing large sets.
Here are the main disadvantages of boosting:
– vulnerability to outliers in data. Boosting models are vulnerable to outliers or
data values that differ from the rest of the data set. Outliers can significantly skew the
results as each model attempts to correct the errors of the previous one;
– implementation in real time. Because this algorithm is more complex than
other processes, real-time boosting can be difficult to implement. Boosting is highly
adaptable, so you can use a variety of model parameters that directly affect its
performance.
Gradient boosting
Python: scikit-learn;
Method: GradientBoostingClassifier, GradientBoostingRegressor.
Gradient boosting is known by many names, including additive regression trees,
stochastic gradient boosting, and gradient boosting. Gradient boosting is a technique
derived from decision trees when Jerome Friedman applied the concept of gradient
boosting to decision trees in 2001. AdaBoost is a variant of gradient boosting. As the
gradient increases, the model is trained sequentially. At each iteration, the model
strives to minimize the loss function (Fig. 8). Similar to random forest decision trees,
gradient boosting is produced from weak predictors. Each new tree is subsequently
trained on data previously misclassified by the previous tree.
This iterative procedure speeds up model training by focusing on more complex
data as simpler predictions are made early in the training phase.
XGBoost is a variation of the gradient boosting method. XGBoost adds
regularization and leverages compute power with distributed multi-threaded processing
to improve speed and efficiency.
40
Fig. 8. Graphical interpretation of the gradient boosting method
Advantages of the gradient boosting method:
– this machine learning algorithm is capable of recovering complex
dependencies;
– the algorithm can be adjusted to any differential quality measure.
Disadvantages of the gradient boosting method:
– the algorithm is prone to overfitting (produces good results on the training set,
but poor results on the test set);
– it is necessary to correctly select the number of classifiers;
– due to the fact that decision trees, which are sensitive to gaps in data, are mainly
used as basic classifiers, the algorithm itself is also sensitive to sparse data.
Adaptive boosting
Python: scikit-learn, NumPy;
Method: AdaBoostClassifier, AdaBoostRegressor.
Adaptive boosting is a popular machine learning technique that iteratively
learns from errors. Decision trees do not learn well by splitting data according to only
one rule. Decision trees are improved by focusing on instances that have been
misclassified. They are identified using weights associated with the data. The model
can learn from its errors and provides a final solution with less bias than a single
decision tree.
Соседние файлы в предмете [НЕСОРТИРОВАННОЕ]