Добавил:
ivanov666
Опубликованный материал нарушает ваши авторские права? Сообщите нам.
Вуз:
Предмет:
Файл:Intelligent data analysis in medicine. Study aid
.pdf
51
CHAPTER 5. METHODS FOR ASSESSING
ARTIFICIAL INTELLIGENCE
Currently, the field of machine learning is still in its infancy. Machine learning
has only been studied in greater detail over the last 25 years, fueling data science to a
large extent. As a result, the data science industry is still in a developmental stage in
front of the endless potential of AI and machine learning. The field of AI solves the
problems of an industry that is accumulating knowledge, experience, and some
problems are emerging for the first time. Generally, the most time-consuming task
within machine learning is determining the appropriate model and the technical
characteristics that significantly influence the final output of the model.
In fact, the features chosen may have a greater impact on the quality of the model
than the model selection itself. Therefore, it is important to correctly evaluate the
learning algorithm that will define the models to predict, classify, and infer the unknown
sample. This is usually done using various metrics, which we will discuss next.
Evaluation of the implemented model
To successfully deploy a machine learning model, you must go through several
development and evaluation phases. The first stage is creating a prototype. In this
phase, a prototype is created by testing different models on early data to determine the
best model. Tuning the hyperparameters of the algorithm is a prerequisite for training
the model. Once the best prototype model is selected, it is tested and verified. Model
validation requires dividing data sets into training, testing, and validation sets. One
must consider the fact that there is no such thing as a random set of data, and instead
randomness is applied to the partitioning of the data set. Be aware of biases that may
appear in the data when splitting this way.
Once the model has been successfully validated, it is deployed to production.
The model is then typically evaluated on one (or more) performance metrics. There are
two ways to evaluate a machine learning model: offline evaluation and live evaluation.
A deployed machine learning model uses data from two sources: historical data
(or data that is used as experience from which to learn) and live data. Many machine
learning models assume that the data distribution is stationary, i.e. the distribution of
52
data is constant over time. However, this is not typical in real life because the
distribution of data often changes over time, which is called distribution shift.
For example, consider a system that predicts drug side effects for patients based
on their health status. Side effects of drugs may vary depending on population factors
such as ethnicity, disease profile, area, climate, etc. The distribution of relevant side
effects based on patient data can change rapidly over time, so it is important for the
model to detect shifts in the distribution and evolve the model accordingly. The way
this is typically viewed is by evaluating the model's performance on live data, assessed
by a metric used in testing and validating the model on historical data. The performance
of a model that is similar to or within the acceptable threshold when evaluated on live
data is considered a model that can be further applied to new data sets.
Deterioration in model performance indicates that the model does not fit the data
and requires retraining. Autonomous evaluation characterizes a model based on metrics
derived and evaluated from a historical, stationary, distributed dataset. A metric such
as accuracy is typically used during the offline learning phase. Offline evaluation
methods include n-fold cross-validation.
Live evaluation refers to the evaluation of performance after the model has been
deployed. The key takeaway is that these metrics may differ from the metrics used to
evaluate performance when the model is deployed in real time. For example, a model
learning new pharmacological treatments may strive to be as accurate as possible
during training and testing, but placement must also consider the cost of the treatment
during deployment. Live assessment, especially in the digital age, can support
multivariate testing to understand the most effective models. Feedback loops are key
to ensuring systems perform as expected and help to better understand the model in the
context of use. This can be performed by a human tester or automated by a contextintelligent agent or by users of the model. It is important that the evaluation of a
machine learning model is based on a statistically independent data set, rather than the
data set on which it is trained. This is because the evaluation of the training data set is
optimistic about the true performance of the model as it adapts to the data set. By
estimating a model with previously unused data, generalization error can be better
assessed. New data is often difficult to find; therefore, it is important to be able to
obtain new data from the current set, i.e. highlight them separately. Often the choice of
data to apply is more important than the choice of machine learning algorithm; and the
better the features used, the better the performance of the model. Many evaluation
metrics can be found in the R language metrics package or scikit-learn for Python.
53
Metrics for assessing the quality of machine learning algorithms
in data classification problems
There are many metrics to evaluate the performance of machine learning tasks.
There are metrics for various machine learning tasks, both classification and
regression. Next, let's look at the most commonly used metrics.
Accuracy.
Accuracy is the simplest metric used to determine how well a model makes
predictions. It is calculated as the percentage of correctly completed forecasts out of
the total number of forecasts made.
Error matrix.
Accuracy is a general metric that does not take into account the separation
between classes. Therefore, it does not address misclassification or associated biases.
For example, a medical misdiagnosis that is a false positive (consider a patient being
diagnosed with breast cancer when she does not have breast cancer) has significantly
different consequences than a false negative, in which the patient is told that she does
not have breast cancer, when in fact it exists [15]. The error matrix breaks down the
correct and incorrect classifications made by the model and assigns them
an appropriate label:
– true positive: the actual class is “yes”, the value of the predicted class is “yes”;
– false positive: the actual class is “no”, and the predicted class is “yes”;
– true negative: the value of the actual class is “no”, and the value of the
predicted class is “no”;
– false negative: the actual class value is “yes” and the predicted class is “no”.
Logarithmic loss.
Logarithmic loss is used for problems in which a continuous probability is
predicted rather than a class label. Logarithmic loss provides a probabilistic measure
of confidence in accuracy and takes into account the entropy between the distribution
of true labels and predictions.
Area under the curve (AUC).
AUC displays the ratio of true positives to the false positive rate. AUC allows
you to visualize the sensitivity and specificity of the classifier. It shows how many
correct positive classifications can be obtained taking into account false positives.
54
Precision, recall, specificity and F-measure.
Precision and recall are two metrics used together to evaluate the performance
of a model. Accuracy evaluates how many items are truly relevant compared to the
total number of items correctly classified. Recall evaluates how many items are
predicted by the model to be relevant out of relevant items.
F-measure goes beyond the arithmetic mean and calculates the harmonic mean
of precision and recall.
Questions for self-control
1. What are the steps involved in deploying a machine learning model?
2. What are the disadvantages of using the “accuracy” metric to assess the quality
of classification in medicine?
3. What is an error matrix? What is it used for?
4. What is the metric for evaluating the log loss classification?
5. What is completeness and F-measure? What is the advantage of using these
metrics?
55
CHAPTER 6. ETHICAL ISSUES IN APPLICATION
OF INTELLIGENT SYSTEMS IN MEDICINE
Ethics or moral philosophy refers to the moral standards of behavior (or set of
moral principles) that shape the behavior of people and the decisions they make.
Morality refers to the principles that distinguish between good/right and bad/wrong
behavior. Ethics in the workplace, for example, are often conveyed through professional
codes of conduct that employees of an institution must abide by.
Data science ethics is a branch of ethics that deals with privacy, decision
making, and data sharing.
Data science ethics includes two main areas:
1) data ethics. This area of data science ethics focuses on the creation, collection,
use, storage, security and transmission of data;
2) ethics of analysis. This area of data science ethics covers the outputs or results
of predictive analytics that data are used to develop.
There are more smartphones in the world than people, and phones, tablets and
digital devices, along with apps, wearables and sensors, create millions of data records
per day. There are currently more than 7 billion mobile phones in use, 112 million
wearable devices sold annually and more than 100,000 health apps available for
download.
Discussions about data storage naturally lead to restrictions on who can see and
process your data. The key is to ensure that only approved services and organizations
have access to your data. Data exchange between applications is commonplace, and
APIs provide accessibility and faster communication between independent services.
For example, users can import nutrition data from various apps into their diabetes or
fitness apps. These services often replicate user data across multiple independent
architectures, leading to challenges in managing approved data access. Applications
that provide data integration must also provide the ability to classify patients’ data.
Systems must be able to validate imported data for data management, auditing, and
patient safety purposes.
Patient data sharing and aggregation of large data sets are used to improve
diagnosis, treatment and care. As data types and quality improve, healthcare accuracy
will improve in areas that require it.
56
Medical datasets with big data can enable predictive analytics to determine
optimal treatment pathways for different patients. Analysis of real-world experience,
clinical studies, randomized clinical trials and pharmacological data facilitates the
search for treatment and management.
Digital health interventions have been shown to improve treatment outcomes for
some diseases. The data has the potential to develop new pharmacological agents and
disrupt traditional treatment paradigms.
The use of patient community-scale evidence, digital education programs, and
health tracking apps is increasingly demonstrating improvements in integral health
outcomes. Concerns about real-world data have typically centered on a lack of
academic reliability. However, such data are increasingly being used in research to
determine optimal strategies for use in populations. Real-world evidence is a vital part
of AI's ethical journey, and there must be trust between patient and provider.
The precise audiences needed for user data and academic research can be
identified more quickly and easily using digital platforms. This provides some benefits
such as faster project hiring time, greater potential, and the ability to compare multiple
sources. Real-world data is used to develop better, more effective medicines. The main
use of machine learning in the short term is in data analytics and predictive analytics.
Safety
There are significant privacy concerns associated with unified systems. What if
such a system is compromised? Do people or patients really need a single system and
the possibility of it being used for evil bigger than the potential benefit? The impact of
vulnerabilities, whether security, operational or technical, is magnified in unified
systems. For years, big companies have been talking about ways to combine and
analyze the giant stores of data they individually collect about people — location data
from smartphones, financial data from banks, relationship data from social media apps,
and search data from browsers — to creating a complete picture of human behavior.
DDoS attacks can wreak havoc on any organization. It is necessary to develop
mitigation procedures and ensure that the network infrastructure provides visibility into
traffic entering and leaving the network. It is good practice to develop a DDoS
protection plan, which should be kept, updated and practiced regularly.
57
The main application of machine learning in healthcare is in the diagnosis and
treatment of patients. AI models are used to help doctors diagnose patients' diseases,
especially in cases involving relatively rare diseases or when treatment outcomes are
difficult to predict. Machine learning ethics refers to the moral issues associated with
the results of machine learning models using data that is associated with users
(patients).
Machine learning has already been used to develop intelligent systems that can
predict mortality risk and life expectancy based on health biomarkers. AI was used to
analyze data to predict the risk of heart failure with high confidence. In addition,
machine learning can be used to determine the most effective drug dosage based on
real-world and clinical patients’ data, reducing healthcare costs for patients and
providers. AI can be used not only to determine dosage, but also to determine the best
suitable drug for a patient. As genetic data becomes available, medications for
conditions such as HIV and diabetes will take into account differences between races,
ethnic groups and individual responses to certain medications. Drug interactions and
side effects can be tracked within the same data. As the technology's limitations are
tested on real patient data, ethical and legal challenges must be overcome.
As AI algorithms become smarter, they also become more complex. Staying
ignorant about building machine learning systems or allowing them to create black
boxes can all lead to ethically problematic results. If an agent were found to be making
incorrect predictions, it would be difficult to determine the behavior that caused the
event, which is hidden and almost impossible to detect. Interpretability of both data
and machine learning models is a critical aspect of intelligent systems. This ensures the
integrity of the model, and the fact that it is attempting to solve the correct problem.
Users of such solutions, including data scientists, will always prioritize results that are
understandable and explainable. Data scientists can also use interpretability metrics as
a basis for model validation and improvement. Neural networks are the most prominent
example of an inexplicable algorithm. The calculated values of the backpropagation
algorithm cannot be explained. As AI becomes more human-like, there will be an
increasing need to ensure that AI does not pick up the bad habits of humans.
Intelligence goes through certain stages of learning. And intelligent systems, like
people, learn from mistakes. Data scientists typically develop machine learning models
with training, testing, and validation steps to ensure that the systems detect the correct
patterns within a certain tolerance. The validation phase of the developed machine
learning model cannot cover all possible permutations of parameters that can be
58
obtained in the real world. These systems can be fooled in ways that humans could not
be fooled. Governance and regular auditing are necessary to ensure that AI systems
work as expected and that humans cannot influence or exploit the model.
Misclassification of predictions can lead to scenarios in which the result is a false
positive or a false negative. The influence of both should be considered in the context
of the problem being solved. Take, for example, misdiagnosis of breast cancer [9]. In
a false positive scenario, the patient will be told that he has breast cancer when he does
not. Knowing that this classification is incorrect will bring some relief to the patient.
A false negative result will lead to progression of the patient's disease and ultimately
to correct rediagnosis. The mental and physical trauma resulting from a false negative
prognosis must be considered. Patients should always be informed of the level of
accuracy of the results. Information management for systems that will be used by
patients (patient-facing), whether prognostic or otherwise, must provide robust
procedures to reduce the risk of error in results. Emotional or psychological patient
support should be considered for those who have experienced trauma as a result of
misdiagnosis.
There is a requirement to ensure the validity of machine learning models over
time to ensure that the model can be generalized and the generalizations are true.
Regular testing and model validation is essential to maintaining the integrity and
accuracy of your machine learning model. A suboptimal predictive analytics model
will produce unreliable results and compromise data integrity.
Ethical analysis poses questions such as whether an autonomous vehicle should
protect its driver or other people; whether a car should primarily protect its occupants
or instead protect other drivers and pedestrians; what if the incident results in loss of
life, what approach should be used then. This may become less of a problem when
autonomous vehicles become more prevalent and involve fewer people.
There are many areas where intelligent health information is used to improve
patients’ health, reduce costs and allocate resources:
– medical services. Patients are increasingly turning to predictive analytics
services to diagnose diseases. For example, diabetic retinopathy is detected using AI
systems;
– pharmacology. New drugs are discovered by studying real-world data and
patient profiles. This allows the pharmaceutical industry to develop new and better
medicines.
59
Health analytics cannot be developed in an environment that ignores society's
ethical requirements for AI, such as respect to personal data. AI ethics should be
integral to the development of systems that can analyze health, identifying agents' risks
before they occur. The ethics of health information is integral to understanding the
purpose of data and the use of results in medicine.
Many organizations have a code of conduct that serves as internal guidance on
expected behavior, requirements and interactions and provides reassurance to external
stakeholders. Codes of conduct should be communicated to employees along with
appropriate training to ensure they are understood and followed. All stakeholders of
the organization must practice and promote compliance with the code of conduct.
A code of ethics is a document used to govern the moral behavior of an
organization. It demonstrates that the organization is committed to responsible business
and technological advancement. A code of ethics describes behavior that an
organization encourages and behavior that is considered harmful to the organization's
own morale, reputation, or clients. It may not cover illegal activities, but it usually
specifies the consequences of non-compliance and how to report such violations.
Employees should be made aware of points that are not obvious to them and then
helped to avoid unintentional but potentially dangerous actions. The code of ethics
should also include a summary of the reasons for using data and its purpose in the
organization's endeavors, and reflect the profitability, integrity and reputation of the
business. It should be free of technical and philosophical jargon and communicate
directly about employee’s expectations. Refer to the code of ethics when exploring the
ethical risks associated with introducing new technologies into your decision-making
process.
When developing a code of ethical conduct from scratch, it is necessary to
consult with employees and stakeholders to understand their views on certain aspects.
Questions to ask:
1. What does AI ethics mean to you?
2. How can our creation improve humanity?
3. What are the potential benefits of what we would like to achieve?
4. How can we improve our code of ethics?
5. Are there any points in the code of ethics that are confusing or require further
clarification?
6. Is a code of ethics useful in decision making?
7. Are the organization's ethics consistent with its employees' own ethical views?
60
Once the various segments of the organization have answered all of the previous
questions, consensus can be reached on an implementation plan. An organization's
code of ethics serves as the starting point for disciplinary action against those who do
not meet standards. The Code of Ethics provides a solid basis for identifying and
addressing ethical issues.
Organizations are responsible for maintaining ethical standards in the collection
and use of patients’ data. Organizations need an approach to data privacy that prevents
breaches and ensures security. Failure to comply with the rules can lead to fines,
reputational consequences and loss of customers. There are various ways to reduce the
risks associated with data collection by taking advantage of the opportunities that big
data can offer.
Questions for self-control
1. What are data science ethics?
2. What are the main areas of data science ethics?
3. What are the ethics of medical data analysis?
4. How important is the issue of security of patient data storage?
5. Why is the code of ethics used in medical organizations?
Соседние файлы в предмете [НЕСОРТИРОВАННОЕ]