Course Overview - Information Theory - Summaries and Lecture materials
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Information Theory
01
Course Overview
TI 2021/2022
Notice
! Author
" João Moura Pires (jmp@fct.unl.pt)
! This material can be freely used for personal or academic purposes without
any previous authorization from the author, provided that this notice is
maintained/kept.
! For commercial purposes the use of any part of this material requires the
previous authorization from the author.
Course Overview - 2
Bibliography
! Many examples are extracted and adapted from:
! And some slides were based on Iain Murray course
" http://www.inf.ed.ac.uk/teaching/courses/it/2014/
Course Overview - 3
Table of Contents
! Context and Motivation
! Let’s code. Repeating Codes
! Let’s code. Block Codes
! What performance the best codes achieve?
! Course Organization and Overview
" Syllabus; Bibliography; Evaluation rules; important dates, etc.
Course Overview - 4
Information Theory
Context and Motivation
Course Overview - 5
A
Boost ‘bo
Analog versus Digital
Communicating Communication
with noise Consider sending an audio signal by amplitude modulation:orit
desired speaker-cone position is the height of the signal. The
ad
shows an encoding of a pure tone.
5 cycles
nea
Signal Econding
A bywith
classical problem amplitude modulation
this type of communication res
channel
attenuation: the amplitude of the signal 100
decayscycles
over time. (T
details of this in a real system could be messy.) Assuming we
Attenuate regularly boost the signal, we would also amplify any noise th
been added to the signal. After several cycles of attenuation,
Add noise addition and amplification, corruption can be severe.
Digital communicationAnalog
A variety of analogue encodings are possible, but whatever Cis
Boost ‘boosting’ process can ever return a corrupted signal exactly
original form. In digital communication the sent message com
Encoding: amplitude
a modulation
discrete set. If the not only
message is choice. we can ‘round’ to t
corrupted
5 cycles
Can re-represent messages to improve
nearest discrete message. Itsignal-to-noise
is possible, but notratio
guaranteed, w
restore the message to exactly the one sent.
100 cycles
Digital encodings: signal takes on discrete values
Digital communication
Digital Communication channels
Signal
Encoding: amplitude modulation not only choice. Corrupted
Can re-represent messages to improve signal-to-noise ratio modem → phone line → modem
digital encoding Recovered
Digital encodings: signal takes on discrete values Galileo → radio waves → Earth
finger tips → nerves → brain
Signal Course Overview - 6
parent cell → daughter cells
an imperfect,
Examples of noisy communication channels
modem ! phone ! modem
line
municate digital
Galileo ! radio ! Earth
orbiting space- waves
daughter
ns information #
"
parent "
cell
cell $
%
$
daughter
cell
nvolve informa- computer ! disk ! computer
n a disk drive, memory drive memory
Course Overview - 7
cross-talk with
General digital communication system
The role ofUniversity
! Cambridge
pyright the encoder
Press is to introduce
2003. systematically
On-screen viewing redundancy
permitted. Printing to make
not permitted. http://www.cam
u can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.
possible to the decoder (which know the encoding process) to discover the sent
1.2: Error-correcting
message codes
even if some bits forflipped
were the binary
by the symmetric channel
noise channel
Source Fi
fo
" co
s ŝ ch
#
in
in
Encoder Decoder de
re
" re
t r so
! Noisy
in
channel
Whereas physical solutions give incremental channel improvements only at
Course Overview - 8
an ever-increasing cost, system solutions can turn noisy channels into reliableLet’s
Let’s consider
consider aa noisy disk drive
noisy disk drive that
that transmits
transmits eacheach bit
bitcorrectly
correctlywith
with
probability
probability (1−f
(1−f )) and incorrectly with
and incorrectly with probability
probability ff. . This
Thismodel
modelcommuni-
communi
Symmetric
cation
cation channel isBinary
channel is known Channel
known as the
as the binary
binary symmetric
symmetricchannel (figure1.4).
channel(figure 1.4).
00 !
!00 P (y = 0 | x = 0)
0) =
= 11 −
−ff;; (y==00| |xx==1)1)
PP(y == ff; ;
xx $$
"#
"
#
"
%
$
yy
"%
$
! P (y = 1 | x = 0)
0) = ff;; (y==11| |xx==1)1) == 11−−f.f.
11 !11 = PP(y
(1 −ff))
(1 −
00 !
! 00
$
$ " #
"#
"
$ "
ff
$
"
"$
" $
" %
$
!
%
$
11 ! 11
(1 −ff))
(1 −
f = 0.1
As an
As an example,
example, let’s
let’s imagine
imagine that
that ff == 0.1,
0.1, that
that is,
is,ten
tenper
percent
centofofthe thebits
bitsare
ar
flipped (figure
flipped (figure 1.5).
1.5). AA useful
useful disk
disk drive
drive would
would flip
flip no
nobits
bitsatatall
allininits
itsentire
entir
lifetime. If we expect to read and write a gigabyte per day for
Course
lifetime. If we expect to read and write a gigabyte per day for ten years, w ten years,
Overview - 9 we
−15Let’s
Let’s consider
consider aa noisy disk drive
noisy disk drive that
that transmits
transmits eacheach bit
bitcorrectly
correctlywith
with
probability
probability (1−f
(1−f )) and incorrectly with
and incorrectly with probability
probability ff. . This
Thismodel
modelcommuni-
communi
Symmetric
cation
cation channel isBinary
channel is known Channel
known as the
as the binary
binary symmetric
symmetricchannel (figure1.4).
channel(figure 1.4).
00 !
!00 P (y = 0 | x = 0)
0) =
= 11 −
−ff;; (y==00| |xx==1)1)
PP(y == ff; ;
xx $$
"#
"
#
"
%
$
yy
"%
$
! P (y = 1 | x = 0)
0) = ff;; (y==11| |xx==1)1) == 11−−f.f.
11 !11 = PP(y
(1 −ff))
(1 −
00 !
! 00
$
$ " #
"#
"
$ "
ff
$
"
"$
" $
" %
$
!
%
$
11 ! 11
(1 −ff))
(1 −
f = 0.1
As an
As an example,
example, let’s
let’s imagine
imagine that
that ff == 0.1,
0.1, that
that is,
is,ten
tenper
percent
centofofthe thebits
bitsare
ar
flipped (figure
flipped (figure 1.5).
1.5). AA useful
useful disk
disk drive
drive would
would flip
flip no
nobits
bitsatatall
allininits
itsentire
entir
lifetime. If we expect to read and write a gigabyte per day for
Course
lifetime. If we expect to read and write a gigabyte per day for ten years, w ten years,
Overview - 10 we
−15Perfect communication over an noisy communication channel?
! A useful disk drive would flip no bits at all in its entire lifetime. If we expect to read
and write a gigabyte per day for ten years, we require a bit error probability of the
order of 10−15, or smaller.
! Physical solutions
! Incremental improvements
! Increasing costs
! System Solutions
! Can turn noisy channels into reliable communication channels (with the only cost being
a computational requirement at the encoder and decoder)
Course Overview - 11Can re-represent messages to improve signal-to-noise ratio mo
General digitalDigital
communication
encodings: signalsystem
takes on discrete values Ga
fi
Signal
p
pyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cam
Corrupted
u can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.
computer mem
Recovered
1.2: Error-correcting codes for the binary symmetric channel
Source Fi
fo
" co
s ŝ ch
#
in
in
Encoder Decoder de
re
" re
t r so
! Noisy
in
channel
Course Overview - 12
Whereas physical solutions give incremental channel improvements only atInformation Theory and Coding Theory
! The role of the encoder is to introduce systematically redundancy to make
pyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cam
possible
u can buy tofor
this book the30decoder (which
pounds or know
$50. See the encoding process) to discover the sent
http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.
message even if some bits
1.2: Error-correcting were
codes forflipped by the symmetric
the binary noise channel.
channel
Source Fi
fo
" co
s ŝ ch
#
in
in
Encoder Decoder de
re
" re
t r so
! Noisy
in
channel
Course Overview - 13
Whereas physical solutions give incremental channel improvements only at
an ever-increasing cost, system solutions can turn noisy channels into reliableInformation Theory and Coding Theory
! The role of the encoder is to introduce systematically redundancy to make
possible to the decoder (which know the encoding process) to discover the sent
message even if some bits were flipped by the noise channel.
! Information Theory is concerned with the theoretical limitations and potentials of
such systems. ‘What is the best error-correcting performance we could
achieve?’
! Coding Theory is concerned with the creation of practical encoding and
decoding systems
Course Overview - 14Information Theory
Course Organization and Overview
Course Overview - 15Syllabus
Course Overview - 16Bibliography
http://www.inference.org.uk/mackay/
http://www.inference.org.uk/mackay/itila/
Course Overview - 17Weekly routine
! Lectures - 1 x 2 h
! The lab sessions - 1 x 2 h
! Training problem solving
! Project developing
! The recommended readings
! The recommended actions
! Meetings for student support if required
Course Overview - 18Evaluation rules
! The students performance evaluation includes two individual written tests and a small
project.
" Final Grade = 35% Test1 + 35% Test2 + 30% Project
! To successful conclude the following constraints are applied:
" Project >= 10;
" Test1 >= 8; Test2 >= 8;
" Average of Test1 and Test2 >= 10;
" Final Grade >= 10.
! The students that get a Project >= 10 and do satisfy the constraints on the tests, may have
an exam which grade will replace the tests in the final grade calculation
Course Overview - 19Web Site: http://ti.ssdi.di.fct.unl.pt
TI - Teoria de Informação
Course Overview - 20Web Site: News
Course Overview - 21Web Site: Information
Course Overview - 23Web Site: Resources
Propose useful stuff
Course Overview - 25Web Site: Summaries
Course Overview - 26Web Site: Information / Schedule
Course Overview - 28Important Dates
! Test 1: Week starting at November 22th
! Test 2: Week starting at January 3th
! Project Specification: Up to 13 November
! Project delivery: Up to end of December
! Project Oral discussion: after 11 January
Course Overview - 29Information Theory
Let’s code. Repeating Codes
Course Overview - 30General
channel digital communication system
ms. What is the
To make
! Athe rules
straightforward idea is to repeat every bit of the message a prearranged number
ct errors; and re-
encode, of
transmit,
times. Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. ht
You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ f
1.2: Error-correcting codes for the binary symmetric channel
Source
Source Transmitted
"
ge a prearranged sequence sequence s ŝ
#
able 1.7. We call s t
Encoder Decoder
0 000
1 111 "
t r
! Noisy
channel
Table 1.7. The repetition code R3 .
The repetition code R3
ng this repetition Whereas physical solutions give incremental channel improvements on
an ever-increasing cost, system solutions can turn noisy channels into re
e vector n to the communication channels with the only cost being a computational require
he binary algebra at the encoder and decoder.
Information theory is concerned with the theoretical limitations an
tor r = t + n are tentials of such systems. ‘What is the best error-correcting performan
could achieve?’
Coding theory is concerned with the creation of practical encoding
decoding systems.
Figure 1.8. An example
Course Overview - 31
transmission using R3 . 1.2 Error-correcting codes for the binary symmetric channel
#$Transmit R3 messages over a BSC
! Transmit R3 messages over a Binary Symmetric Channel with f = 0.1
! We can describe the channel as ‘adding’ a sparse noise vector n to the transmitted
vector t (adding in modulo 2 arithmetic):
! A zero onCopyright
n does not change
Cambridge the
University Press transmitted
2003. bit
On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981
You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.
! A one on n does1.2:
change the transmitted
Error-correcting bit 0symmetric
codes for the binary → 1; 1channel
→0 5
Source Figure 1.6. The ‘system’ solution
for achieving reliable
" communication over a noisy
s ŝ channel. The encoding system
#
introduces systematic redundancy
into the transmitted vector t. The
Encoder Decoder decoding system uses this known
redundancy to deduce from the
" received vector r both the original
t r source vector and the noise
! Noisy
introduced by the channel.
channel
r = t ⊕n
Whereas physical solutions give incremental channel improvements only at
an ever-increasing cost, system solutions can turn noisy channels into reliable
communication channels with the only cost being a computational requirement Course Overview - 32
at the encoder and decoder.
Information theory is concerned with the theoretical limitations and po-straightforward idea is to repeat every bit of the message a prearranged
smber of times – for example, three times, as shown in table 1.7. We call
Transmit R3 messages over a BSC Source T
is repetition code ‘R3 ’. Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://w
d idea is to repeat every bit of the You can message a prearranged
buy this book for 30 pounds sequence for lin
or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/
Imagine that we transmit the source1.2:message s
– for example, three times, as shown in Error-correcting table 1.7.codesWe for thecall
binary symmetric channel
! Lets send a message with few bits
ode ‘R3 ’.
s=0 0 1 0 1 1 0 Source
0
we transmit the source message s
"
ŝ 1
#
er a binary symmetric channel with noise level Encoder f = 0.1 using this Decoderrepetition
s=0 0 1 0 1 1 0
de. We can describe the channel as ‘adding’ a sparse noise vector Table
"
n1.7.to Thethe r
ansmitted vector – adding in modulo 2 arithmetic,
mmetric channel with noise level f = 0.1 using this repetition
t
! i.e., the
Noisy binary r
algebra
channel
which the
scribe 1+1=0.
channelA possible noise
as ‘adding’ vectornoise
a sparse n and vectorreceived n to vector
the r = t + n are
own
or –! in figurein1.8.
adding modulo 2 arithmetic, i.e., the
Whereas binary
physical algebra
solutions give incremental channel improvements only a
The transmitted message according to different noisecost,
an ever-increasing vectors
system solutions can turn noisy channels into reliabl
. A possible noise vector n and receivedatcommunication
vector r =
channels
the encoder and decoder.
t +
with n
the are
only cost being a computational requiremen
1.8. Information theory is concerned with the theoretical limitations and po
s 0 0 1 0tentials 1
of such
could achieve?’
systems. 1 ‘What is the0best error-correcting performance w
!"#$ !"#$ !"#$ !"#$ !"#$ !"#$ !"#$
1 1 1 1 1 0 0 0 Figure 1.8. An e
Coding theory is concerned with the creation of practical encoding an
t 0 0 0 0 0 0 1 1 1 0 0decoding 0 1systems.
0 0 n 10 0 0 00 0 1 10 0 0 10 0 0 0 1 0 1 0 0 0 0 0 0 transmission usi
!"#$ !"#$ !"#$ !"#$ !"#$ !"#$ 1.2!"#$ Error-correcting codes for the binary symmetric channel
0 0 0 0 0 0 r 1 1010 00 0000 11 1111 11 1010We00now0 consider
0 1 0 examples 1 1 of1 encoding
0 0 and0 decoding systems. What is th
0 0 0 0 0 1 0 0 0 0 0 0 1 0 1 0 0 0 simplest 0 0 way to add useful redundancy to a transmission? [To make the rule
0
of the game clear: we want to be able to detect and correct errors; and re
0 0 0 0 0 1 1 1 1 0 0 0 0 1 0 1 1 1 transmission0 0 0
and decode.]
is not an option. We get only one chance to encode, transmi
Course Overview - 33
How should we decode this received Repetition vector? codesThe optimal algorithm looksstraightforward idea is to repeat every bit of the message a prearranged
smber of times – for example, three times, as shown in table 1.7. We call
Transmit R3 messages over a BSC Source T
is repetition code ‘R3 ’. Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://w
d idea is to repeat every bit of the You can message a prearranged
buy this book for 30 pounds sequence for lin
or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/
Imagine that we transmit the source1.2:message s
– for example, three times, as shown in Error-correcting table 1.7.codesWe for thecall
binary symmetric channel
! Lets send a message with few bits
ode ‘R3 ’.
s=0 0 1 0 1 1 0 Source
0
we transmit the source message s
"
ŝ 1
#
er a binary symmetric channel with noise level Encoder f = 0.1 using this Decoderrepetition
s=0 0 1 0 1 1 0
de. We can describe the channel as ‘adding’ a sparse noise vector Table
"
n1.7.to Thethe r
ansmitted vector – adding in modulo 2 arithmetic,
mmetric channel with noise level f = 0.1 using this repetition
t
! i.e., the
Noisy binary r
algebra
channel
which the
scribe 1+1=0.
channelA possible noise
as ‘adding’ vectornoise
a sparse n and vectorreceived n to vector
the r = t + n are
own
or –! in figurein1.8.
adding modulo 2 arithmetic, i.e., the
Whereas binary
physical algebra
solutions give incremental channel improvements only a
The transmitted message according to different noisecost,
an ever-increasing vectors
system solutions can turn noisy channels into reliabl
. A possible noise vector n and receivedatcommunication
vector r =
channels
the encoder and decoder.
t +
with n
the are
only cost being a computational requiremen
1.8. Information theory is concerned with the theoretical limitations and po
s 0 0 1 0tentials 1
of such
could achieve?’
systems. 1 ‘What is the0best error-correcting performance w
!"#$ !"#$ !"#$ !"#$ !"#$ !"#$ !"#$
1 1 1 1 1 0 0 0 Figure 1.8. An e
Coding theory is concerned with the creation of practical encoding an
t 0 0 0 0 0 0 1 1 1 0 0decoding 0 1systems.
0 0 n 10 0 0 00 0 1 10 0 0 10 0 0 0 1 0 1 0 0 0 0 0 0 transmission usi
!"#$ !"#$ !"#$ !"#$ !"#$ !"#$ 1.2!"#$ Error-correcting codes for the binary symmetric channel
0 0 0 0 0 0 r 1 1010 00 0000 11 1111 11 1010We00now0 consider
0 1 0 examples 1 1 of1 encoding
0 0 and0 decoding systems. What is th
0 0 0 0 0 1 0 0 0 0 0 0 1 0 1 0 0 0 simplest 0 0 way to add useful redundancy to a transmission? [To make the rule
0
of the game clear: we want to be able to detect and correct errors; and re
0 0 0 0 0 1 1 1 1 0 0 0 0 1 0 1 1 1 transmission0 0 0
and decode.]
is not an option. We get only one chance to encode, transmi
Course Overview - 34
How should we decode this received Repetition vector? codesThe optimal algorithm looksstraightforward idea is to repeat every bit of the message a prearranged
smber of times – for example, three times, as shown in table 1.7. We call
Transmit R3 messages over a BSC Source T
is repetition code ‘R3 ’. Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://w
d idea is to repeat every bit of the You can message a prearranged
buy this book for 30 pounds sequence for lin
or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/
Imagine that we transmit the source1.2:message s
– for example, three times, as shown in Error-correcting table 1.7.codesWe for thecall
binary symmetric channel
! Lets send a message with few bits
ode ‘R3 ’.
s=0 0 1 0 1 1 0 Source
0
we transmit the source message s
"
ŝ 1
#
er a binary symmetric channel with noise level Encoder f = 0.1 using this Decoderrepetition
s=0 0 1 0 1 1 0
de. We can describe the channel as ‘adding’ a sparse noise vector Table
"
n1.7.to Thethe r
ansmitted vector – adding in modulo 2 arithmetic,
mmetric channel with noise level f = 0.1 using this repetition
t
! i.e., the
Noisy binary r
algebra
channel
which the
scribe 1+1=0.
channelA possible noise
as ‘adding’ vectornoise
a sparse n and vectorreceived n to vector
the r = t + n are
own
or –! in figurein1.8.
adding modulo 2 arithmetic, i.e., the
Whereas binary
physical algebra
solutions give incremental channel improvements only a
The transmitted message according to different noisecost,
an ever-increasing vectors
system solutions can turn noisy channels into reliabl
. A possible noise vector n and receivedatcommunication
vector r =
channels
the encoder and decoder.
t +
with n
the are
only cost being a computational requiremen
1.8. Information theory is concerned with the theoretical limitations and po
s 0 0 1 0tentials 1
of such
could achieve?’
systems. 1 ‘What is the0best error-correcting performance w
!"#$ !"#$ !"#$ !"#$ !"#$ !"#$ !"#$
1 1 1 1 1 0 0 0 Figure 1.8. An e
Coding theory is concerned with the creation of practical encoding an
t 0 0 0 0 0 0 1 1 1 0 0decoding 0 1systems.
0 0 n 10 0 0 00 0 1 10 0 0 10 0 0 0 1 0 1 0 0 0 0 0 0 transmission usi
!"#$ !"#$ !"#$ !"#$ !"#$ !"#$ 1.2!"#$ Error-correcting codes for the binary symmetric channel
0 0 0 0 0 0 r 1 1010 00 0000 11 1111 11 1010We00now0 consider
0 1 0 examples 1 1 of1 encoding
0 0 and0 decoding systems. What is th
0 0 0 0 0 1 0 0 0 0 0 0 1 0 1 0 0 0 simplest 0 0 way to add useful redundancy to a transmission? [To make the rule
0
of the game clear: we want to be able to detect and correct errors; and re
0 0 0 0 0 1 1 1 1 0 0 0 0 1 0 1 1 1 transmission0 0 0
and decode.]
is not an option. We get only one chance to encode, transmi
Course Overview - 35
How should we decode this received Repetition vector? codesThe optimal algorithm looksstraightforward idea is to repeat every bit of the message a prearranged
smber of times – for example, three times, as shown in table 1.7. We call
Transmit R3 messages over a BSC Source T
is repetition code ‘R3 ’. Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://w
d idea is to repeat every bit of the You can message a prearranged
buy this book for 30 pounds sequence for lin
or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/
Imagine that we transmit the source1.2:message s
– for example, three times, as shown in Error-correcting table 1.7.codesWe for thecall
binary symmetric channel
! Lets send a message with few bits
ode ‘R3 ’.
s=0 0 1 0 1 1 0 Source
0
we transmit the source message s
"
ŝ 1
#
er a binary symmetric channel with noise level Encoder f = 0.1 using this Decoderrepetition
s=0 0 1 0 1 1 0
de. We can describe the channel as ‘adding’ a sparse noise vector Table
"
n1.7.to Thethe r
ansmitted vector – adding in modulo 2 arithmetic,
mmetric channel with noise level f = 0.1 using this repetition
t
! i.e., the
Noisy binary r
algebra
channel
which the
scribe 1+1=0.
channelA possible noise
as ‘adding’ vectornoise
a sparse n and vectorreceived n to vector
the r = t + n are
own
or –! in figurein1.8.
adding modulo 2 arithmetic, i.e., the
Whereas binary
physical algebra
solutions give incremental channel improvements only a
The transmitted message according to different noisecost,
an ever-increasing vectors
system solutions can turn noisy channels into reliabl
. A possible noise vector n and receivedatcommunication
vector r =
channels
the encoder and decoder.
t +
with n
the are
only cost being a computational requiremen
1.8. Information theory is concerned with the theoretical limitations and po
s 0 0 1 0tentials 1
of such
could achieve?’
systems. 1 ‘What is the0best error-correcting performance w
!"#$ !"#$ !"#$ !"#$ !"#$ !"#$ !"#$
1 1 1 1 1 0 0 0 Figure 1.8. An e
Coding theory is concerned with the creation of practical encoding an
t 0 0 0 0 0 0 1 1 1 0 0decoding 0 1systems.
0 0 n 10 0 0 00 0 1 10 0 0 10 0 0 0 1 0 1 0 0 0 0 0 0 transmission usi
!"#$ !"#$ !"#$ !"#$ !"#$ !"#$ 1.2!"#$ Error-correcting codes for the binary symmetric channel
0 0 0 0 0 0 r 1 1010 00 0000 11 1111 11 1010We00now0 consider
0 1 0 examples 1 1 of1 encoding
0 0 and0 decoding systems. What is th
0 0 0 0 0 1 0 0 0 0 0 0 1 0 1 0 0 0 simplest 0 0 way to add useful redundancy to a transmission? [To make the rule
0
of the game clear: we want to be able to detect and correct errors; and re
0 0 0 0 0 1 1 1 1 0 0 0 0 1 0 1 1 1 transmission0 0 0
and decode.]
is not an option. We get only one chance to encode, transmi
Course Overview - 36
How should we decode this received Repetition vector? codesThe optimal algorithm looksHow should we decode this received vector?
! The optimal algorithm looks at the received bits three at a time and takes a majority
vote.
opyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642
ou can buy!thisMore 0, 30
book for take a 0 or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.
pounds
!6 More 1s, take a 1 1 — Introduction to Inform
Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted.
You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila
Algorithm 1.9. Maj
Received sequence r Likelihood ratio = 1)Error-correcting
P (r | s 1.2:
Decoded codessequence ŝ
for the binary decoding
symmetric algorithm
channel
P (r | s = 0)
shown are the likel
000 γ −3 Source
0 (1.23), assuming th
001 γ −1 0 binary
" symmetric
s ŝ
010 γ −1 #0 γ ≡ (1 − f )/f .
100 γ −1 0
Encoder Decoder
101 γ1 1
110 γ1 t 1
"
r
Noisy
011 γ1 1! channel
111 γ3 1
Whereas physical solutions give incremental channel improvements
an ever-increasing cost, system solutions can turn noisy channels into r
communication channels with the only cost being a computational requi
At the risk of explaining the obvious, let’s prove this result.
at the encoder The optimal
and decoder. Course Overview - 37
decoding decision (optimal in the sense of having the smallest probabilitywith
Information theory is concerned of the theoretical limitations a
being wrong) is to find which tentials
value of s is most of such systems.
probable, given r.‘What is the best error-correcting performa
ConsiderHow should we decode this received vector?
! The optimal algorithm looks at the received bits three at a time and takes a majority
vote.
opyright Cambridgeviewing
s 2003. On-screen University Press 2003.
permitted. On-screen
Printing viewinghttp://www.cambridge.org/0521642981
not permitted. permitted. Printing not permitted. http://www.cambridge.org/0521642
ou
or can
$50.buy
See More
!this 0, 30
book for take a 0 or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/
pounds
http://www.inference.phy.cam.ac.uk/mackay/itila/ for links. for links.
!6 More 1s, take a 1 1 — Theory
1 — Introduction to Information Introduction to Inform
Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted.
You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila
Algorithm 1.9. Majority-vote Algorithm 1.9. Maj
P (r | s = 1) | s = 1)Error-correcting codes for the binary symmetric channel
ce r Received
Likelihood ratiosequence r
P (r | s = 0)
Likelihood
Decoded ratio PP (r
sequence Decoded
= 0) decoding
ŝ(r | s 1.2: sequence
algorithm forŝR3 . Alsodecoding algorithm
shown are the likelihood ratios shown are the likel
γ −3000 0 γ −3 Source
0 the channel is
(1.23), assuming (1.23),
a assuming th
γ −1001 0 γ −1 binary symmetric
0 channel; binary
" symmetric
s ŝ
γ −1010 0 γ −1 γ ≡ (1 − f )/f
0.
# γ ≡ (1 − f )/f .
γ −1100 0 γ −1 0
Encoder Decoder
γ 1 101 1 γ1 1
γ 1 110 1 γ1 t 1 "
r
Noisy
γ 1 011 1 γ1 1! channel
γ 3 111 1 γ3 1
Whereas physical solutions give incremental channel improvements
an ever-increasing cost, system solutions can turn noisy channels into r
communication channels with the only cost being a computational requi
explaining the At the risk
obvious, let’sofprove
explaining the obvious,
this result. let’s prove
The optimal this result.
at the encoder The optimal
and decoder. Course Overview - 38
on (optimal in decoding
the decision
sense of having(optimal in the
the smallest sense of having
probability of the smallest probabilitywith
Information theory is concerned of the theoretical limitations a
to find which beingofwrong)
value is toprobable,
s is most find which value
given r. ofConsider tentials
s is most of such systems.
probable, given r.‘What is the best error-correcting performa
ConsiderHow should we decode this received vector?
! The optimal algorithm looks at the received bits three at a time and takes a majority
vote.
! The optimal decoding decision (optimal in the sense of having the smallest
Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981
y this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.
probability
1.2: Error-correcting codesof
for being wrong)
the binary symmetricischannel
to find which value of s is most probable,
5 given r.
Source Figure 1.6. The ‘system’ solution
for achieving reliable
"
s
#
ŝ Posteriori probability of s
communication over a noisy
channel. The encoding system
introduces systematic redundancy
into the transmitted vector t. The
Encoder Decoder decoding system uses this known
P(r r r | s)P(s)
redundancy to deduce from the
" P(s | r r r ) =
received vector
1 2 3r both the
1 2 3
original
t
! Noisy
r P(r r r )
source vector and the noise
introduced by the channel. 1 2 3
channel r = (r1r2 r3 )
Whereas physical solutions give incremental channel improvements only at
an ever-increasing cost, system solutions can turn noisy channels into reliable
communication channels with the only cost being a computational requirement
at the encoder and decoder.
Information theory is concerned with the theoretical limitations and po-
tentials of such systems. ‘What is the best error-correcting performance we
could achieve?’ Course Overview - 39
Coding theory is concerned with the creation of practical encoding and
decoding systems.How should we decode this received vector?
! Which value of s is most probable, given r.
P(r1r2 r3 | s = 1)P(s = 1) P(r1r2 r3 | s = 0)P(s = 0)
P(s = 1 | r1r2 r3 ) = P(s = 0 | r1r2 r3 ) =
P(r1r2 r3 ) P(r1r2 r3 )
! Depends on
! prior probability P(s)
! the data-dependent term P(r1r2r3 | s) - the likelihood of s
! The normalizing constant P(r1r2r3) does not needs to be calculated to find the most
probable s given the received sequence
Course Overview - 40How should we decode this received vector?
! Which value of s is most probable, given r.
P(r1r2 r3 | s = 1)P(s = 1) P(r1r2 r3 | s = 0)P(s = 0)
P(s = 1 | r1r2 r3 ) = P(s = 0 | r1r2 r3 ) =
P(r1r2 r3 ) P(r1r2 r3 )
! Assumptions:
! We assume that the prior probability are equal: P(s = 0) = P(s = 1) = 0.5
! We assume that the channel is a Binary Symmetric Channel with noise level f < 0.5
! So maximizing P(s | r1r2r3) just requires to maximize the the likelihood P(r | s)
N
N = 3 is the number of bits in the block
P(r | s) = P(r | t(s)) = ∏ P(rn | t n (s))
n=1
The BSC has no memory !
Course Overview - 41How should we decode this received vector?
! So maximizing P(s | r1r2r3) just requires to maximize the the likelihood P(r | s)
N N = 3 is the number of bits in the block
P(r | s) = P(r | t(s)) = ∏ P(rn | t n (s))
n=1 The BSC has no memory !
! Where P(rn | tn),
⎧
⎪ (1− f ) if rn = t n
P(rn | t n ) = ⎨
⎪ f if rn ≠ t n
⎩
! and the likelihood ratio for the two possible hypotheses is,
P(r | s = 1) N
P(rn | t n (1))
P(rn |t n (1))
P(rn |t n (0)) = 1− f
f if rn = 1
=∏
P(r | s = 0) n=1 P(rn | t n (0))
P(rn |t n (1))
P(rn |t n (0)) = f
1− f if rn = 0
Course Overview - 42How should we decode this received vector?
! The likelihood ratio for the two possible hypotheses is,
P(r | s = 1) N
P(rn | t n (1))
P(rn |t n (1))
P(rn |t n (0)) = 1− f
f if rn = 1
=∏
P(r | s = 0) n=1 P(rn | t n (0))
P(rn |t n (1))
P(rn |t n (0)) = f
1− f if rn = 0
! Since f < 0.5, γ = >1
1− f
f and so,
! The winning hypothesis is the one with the most ‘votes’.
! Each vote counting for a factor of γ in the likelihood ratio.
Course Overview - 43pyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambr
How should we decode this received vector?
u can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.
6 1 — Introduc
! The majority vote to decode R3.
Alg
P (r | s = 1)
Received sequence r Likelihood ratio P (r | s = 0) Decoded sequence ŝ dec
sho
000 γ −3 0 (1.2
001 γ −1 0 bin
010 γ −1 0 γ≡
100 γ −1 0
101 γ1 1
110 γ1 1
011 γ1 1
111 γ3 1
At the risk of explaining the obvious, let’s prove this result. The optimal
decoding decision (optimal in the sense of having the smallest probability of
being wrong) is to find which value of s is most probable, given r. Consider
the decoding of a single bit s, which was encoded as t(s) and gave rise to three
received bits r = r1 r2 r3 . By Bayes’ theorem, the posterior probability of s is -
Course Overview 44We now apply
Repetition the majority vote decoder to the received vector of figure 1.8.
codes
Decoding a received
The first three message bits are all 0, so we decode this triplet as a 0. In the Sou
A straightforward
second idea
triplet of figure 1.8, is to
there are repeat every
two 0s and bitsoof
one 1, wethe message
decode a prearranged
this triplet sequ
asnumber of times
a 0 – which in this– case
for example, three
corrects the error.times,
Not all as errors
shownareincorrected,
table 1.7. We call
The
this optimal
! however. If we algorithm
are code
repetition unlucky looks
‘R at the
3 ’. two
and received
errors fall in abits three
single at aastime
block, and
in the takes a majority
fifth
tripletImagine
of figure that
1.8, then the decoding
we transmit therule gets message
source the wrong answer, as shown
invote.
figure 1.10.
s=0 0 1 0 1 1 0
Figure 1.10. Decodin
Table
s
0 0 1 0 1 1 0 vector from figure 1.
over a binary symmetric
!"#$channel
!"#$ with
!"#$ noise
!"#$ level = 0.1!"#$
!"#$ f !"#$ using this repetition
t 000 000 111 000 111 111 000
code. We can describe the channel as ‘adding’ a sparse noise vector n to the
n 000 001 000 000 101 000 000
transmitted vector r– adding
0 0 0 0 in1 modulo
0 1 1 1 0 20 0arithmetic,
0 1 0 1 1 1i.e.,
0 0 the
0 binary algebra
#$!" #$!" #$!" #$!" #$!" #$!" #$!"
in which 1+1=0. Aŝ possible
0
noise 1vector0 n and
0 0
received
1
vector
0
r = t + n are
shown corrected
in figureerrors
1.8. !
undetected errors !
Figure
s 0 0 1 0 1 1 0 trans
! no errors: the message !"#$is correctly
!"#$ !"#$ !"#$ !"#$ !"#$ !"#$
decoded
t 000 000 111 000 111 111 000
Exercise 1.2.[2, p.16]nShow
0 0 that
0 0the
0 1 error
0 0 probability
0 0 0 0 is0reduced
1 1 0 0 by the
0 0 0 use of
0 The exercise’s rating
! one error: the original is recovered indicates its difficult
R3 by computing the error probability of this code for a binary symmetric
r 000 001 111 000 010 111 000 exercises are the easi
channel with noise level f .
! two or three errors: the message is incorrectly decoded. that are accompanie
marginal rat are esp
The error probability is dominated by the probability that two bits in
recommended. If a s
How
a block should
of three arewe decode
flipped, thisscales
which received
as fvector?
2 . In theThe
caseoptimal algorithm
of the binary looks
Coursepartial solution
Overview - 45 is pr
symmetric channel bits
at the received with three
f = 0.1,
at the R 3 code
a time andhas a probability
takes a majority of vote
error,(algorithm
after page1.9).
is indicated aftProbability of error of R3 coding?
! An error is made by R3 if two or more bits are flipped in a block of three.
! The error probability of R3 is a sum of two terms:
! the probability that all three bits are flipped = f 3; Pb = 3 f (1− f ) + f
2 3
! the probability that exactly two bits are flipped, 3f 2(1 − f). Pb = 3 f 2 − 2 f 3
Pb 3f 2
f = 0.1
any error
2 bits errors
3 bits errors
Pb = 0.03
f
Course Overview - 46that are accompanied by
marginal rat are especial
The error probability is dominated by the probability that two bits in
recommended. If a soluti
The improvement on Pb with R3
a block of three are flipped, which scales as f 2 . In the case of the binary partial solution is provid
symmetric channel with f = 0.1, the R 3 code has a probability of error, after page is indicated after th
decoding, of pb ! 0.03 per bit. Figure 1.11 shows the result of transmitting a difficulty rating; for exam
binary image over a binary symmetric channel using the repetition code. exercise’s solution is on p
! Assuming a BSC with f = 0.1
s encoder t channel r decoder ŝ
f = 10%
! ! !
f = 0.1
Figure 1.11. Transmittin
source bits over a binary
symmetric channel with
using a repetition code a
majority vote decoding a
The probability of decod
error has fallen to about
Prate has fallen to 1/3.
b = 0.03
Course Overview - 47The improvement on Pb with R3: What is the cost?
! Assuming a BSC with f = 0.1, with R3 we reduce the error probability from 0.1 to
0.03, but.
! The rate of information transfer has fallen by a factor of three
Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981
!You We would need three of the original noisy gigabyte disk drives in order to create a one-
can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.
gigabyte
8 disk drive with pb = 0.03 ! 1 — Introduction to Information Theory
Figure 1.12. Error probability pb
0.1
0.1 R1 0.01 R5 R1 versus rate for repetition codes
R3
over a binary symmetric channel
with f = 0.1. The right-hand
0.08
1e-05 more useful codes figure shows pb on a logarithmic
pb scale. We would like the rate to
0.06 be large and pb to be small.
0.04 1e-10
R3
0.02
R5 more useful codes
R61 R61
0 1e-15
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1
Rate Rate
Course Overview - 48
The repetition code R3 has therefore reduced the probability of error, as
desired. Yet we have lost something: our rate of information transfer hasThe improvement on Pb with R3: What is the cost?
! Assuming a BSC with f = 0.1, with R3 we reduce the error probability from 0.1 to
0.03, but.
! The rate of information transfer has fallen by a factor of three
Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981
!You We would need three of the original noisy gigabyte disk drives in order to create a one-
can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.
gigabyte
8 disk drive with pb = 0.03 ! 1 — Introduction to Information Theory
Figure 1.12. Error probability pb
0.1
0.1 R1 0.01 R5 R1 versus rate for repetition codes
R3
over a binary symmetric channel
with f = 0.1. The right-hand
0.08
1e-05 more useful codes figure shows pb on a logarithmic
pb scale. We would like the rate to
be large and pb to be small.
Log scale
0.06
0.04 1e-10
R3
0.02
R5 more useful codes
R61 R61
0 1e-15
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1
Rate Rate
Course Overview - 49
The repetition code R3 has therefore reduced the probability of error, as
desired. Yet we have lost something: our rate of information transfer hasWhat improvements could we expect? At What rate?
! Can we push the error probability lower, to the values required for a sellable disk
drive (e.g. 10−15) ?
! So to build a single gigabyte disk drive with the required reliability from noisy
Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981
You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.
gigabyte drives with f = 0.1, we would need 60 of the noisy disk drives
8 1 — Introduction to Information Theory
Figure 1.12. Error probability pb
0.1
0.1 R1 0.01 R5 R1 versus rate for repetition codes
R3
over a binary symmetric channel
with f = 0.1. The right-hand
0.08
1e-05 more useful codes figure shows pb on a logarithmic
pb scale. We would like the rate to
be large and pb to be small.
Log scale
0.06
0.04 1e-10
R3
0.02
R5 more useful codes
R61 R61
0 1e-15
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1
Rate Rate
The repetition code R3 has therefore reduced the probability of error, as
desired. Yet we have lost something: our rate of information transfer has Course Overview - 50
fallen by a factor of three. So if we use a repetition code to communicate dataInformation Theory
Block codes – the (7, 4) Hamming code
Course Overview - 51Block Codes
! Add redundancy to blocks of data instead of encoding one bit at a time
! A block code is a rule for converting a sequence of source bits s, of length K, say,
into a transmitted sequence t of length N bits.
! To add redundancy, N > K
K bits N bits
s Encoder t
! In a linear block code, the extra N − K bits are linear functions of the original K bits
K bits N bits
s Encoder t
s parity-check
Course Overview - 52You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam
Hamming Code (7, 4) 1.2: Error-correcting codes for the binary symmetric ch
! Linear block code with N = 7; K = A
4 block code is a rule for converting a sequence of
7 bits
K, say, into a transmitted sequence t of length N bits
! 4 source bits t
we make N greater than K. In a linear block code, the
s1 s2 s3 s4 t5 t5 t5
! 3 parity check bits linear functions of the original K bits; these extra bits
bits. An example of a linear block code is the (7, 4)
! The 3 parity check bits are linear combinations of the message bits
transmits N = 7 bits for every K = 4 source bits.
t5 1
t 5 = s1 ⊕ s2 ⊕ s3
s1 s2
t 6 = s2 ⊕ s 3 ⊕ s 4 s3
1
0
t 7 = s1 ⊕ s3 ⊕ s4 t7 s4 t
6
1 0
(a) (b)
The encoding operation for the code is shown pictor
arrange the seven transmitted bits in three intersecting
transmitted bits, t1 t2 t3 t4 , are set equal
Course to the four
Overview - 53 sou
parity-check bits t5 t6 t7 are set so that the parity withK, say, into a transmitted sequence t of length N bits. To add redundancy,
we make N greater
Hamming Codethan K. In a linear block code, the extra N − K bits are
(7, 4)
linear functions of the original K bits; these extra bits are called parity-check
bits. An example of a linear block code is the (7, 4) Hamming code, which
! Transmitting
transmits = 1000
N = 7s bits for every K = 4 source bits.
Figu
t5 1 repr
(7, 4
s1 s2 1 0
s3 0
t7 s4 t 1 0
6 0
(a) (b)
The encoding operation for the code is shown pictorially in figure 1.13. We
arrange thetseven
5 = s 1 ⊕ s 2 ⊕
transmitted s 3 t 5 =
bits in three intersecting 1⊕ 0 ⊕
circles. The
0 first four
transmitted bits, t1 t2 t3 t4 , are set equal to the four source bits, s 1 s2 s3 s4 . The
= st25⊕
parity-checkt 6bits ⊕ sset
t6 t7s3are 6 = 0⊕
4 so that the parity twithin 0 ⊕circle
each 0 is even:
the first parity-check bit is the parity of the first three source bits (that is, it
t 7 = s 1 ⊕ s 3 ⊕ s 4 t 7 = 1⊕ 0 ⊕ 0
is 0 if the sum of those bits is even, and 1 if the sum is odd); the second is
the parity of the last three; and the third parity bit is the parity of source bits
one, three and four. Course Overview - 54
As an example, figure 1.13b shows the transmitted codeword for the casethe parity of the last three; and the third parity bit is the parity of source bits
one, three and four.
Hamming Code (7,
As an example, 4): 1.13b
figure codes for the
shows thetransmitted
24 messages codeword for the case
s = 1000. Table 1.14 shows the codewords generated by each of the 2 4 =
sixteen settings of the four source bits. These codewords have the special
property that any pair differ from each other in at least three bits.
s t s t s t s t Tab
{t}
0000 0000000 0100 0100110 1000 1000101 1100 1100011 An
0001 0001011 0101 0101101 1001 1001110 1101 1101000 eac
0010 0010111 0110 0110001 1010 1010010 1110 1110100
0011 0011100 0111 0111010 1011 1011001 1111 1111111
Because the Hamming code is a linear code, it can be written compactly in
! In H(7, 4) any
terms pair of codewords
of matrices as follows.differ
The from each other
transmitted in at least
codeword three bits
t is obtained from the
source sequence s by a linear operation,
! What this suggest in terms of its capabilities of detecting errors?
t = GTs, (1.25)
! Or even in terms of its capabilities of correcting errors?
where G is the generator matrix of the code,
1 0 0 0
0 1 0 0
0 0 1 0 Course Overview - 55
G =
T
0 0 0 1 ,
(1.26)Decoding the Hamming Code (7, 4)
! We assume a Binary Symmetric Channel and that all source vectors are
equiprobable.
! The optimal decoder identifies the source vector s whose encoding t(s) differs
from the received vector r in the fewest bits.
! This corresponds to find the closest codeword of r.
! Since any pair of codewords differ from each other in at least three bits, the H(7, 4)
will detect and correct any error on a single bit.
! It will be misleading with errors on two bits.
! Each error on one bit is associated to a syndrome.
Course Overview - 56ht Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/05216429
n buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.
Decoding the Hamming Code (7, 4)
1.2: Error-correcting codes for the binary symmetric channel
Figure 1.15. Pictoria
r5 Which circles violate the s = 1000representation of dec
parity check? Hamming (7, 4) code
r1 r2 received vector is wr
t = 1000101
r3 The circles associated to diagram as shown in
t5 and t6 (b,c,d,e), the receive
r7 r4 r
6
n = 0100000
shown, assuming tha
(a) transmitted vector w
Which bits are involved figure 1.13b and the
r = 1100101
1 *
in all 0circles with a 1 by ! were flipped. T
parity checks are hig
violation?
1 1* 1 0 1 0 dashed circles. One
0 0 1* bits is the most prob
only r2 ! (the flipped bit) to account for each ‘
1 0 0 1 0 0 1 0 i.e., each pattern of
0
(b) (c) (d) satisfied parity check
The syndrome to this t 5 = sIn
1 ⊕ s2 ⊕ s 3
examples (b), (c),
most probable suspe
error is based on the t 6 = sbit
2 ⊕that
s3 ⊕was
s4 flipped.
1 parity of the circles
1 In example (e), two
t 7 = sflipped,
1 ⊕ s3 ⊕ s3sand
4 t7 . T
1 0 z = (110) 1 1 probable suspect is r
!
1* 1* a circle in (e! ), which
output of the decodi
0* 0 0 0* 0 0 Course Overview - 57
(e) (e! )ht Cambridge
2003. University
On-screen viewingPress 2003. Printing
permitted. On-screennotviewing permitted.
permitted. Printing not permitted. http://www.cambridge.org/05216429
http://www.cambridge.org/0521642981
nr buy
$50.this
Seebook for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/
http://www.inference.phy.cam.ac.uk/mackay/itila/ for links. for links.
Decoding the Hamming Code (7, 4)
1.2: Error-correcting codes for the binary symmetric channel
codes for the binary symmetric channel 11
Figure 1.15. Pictoria
r5 Which circles violate the1.15. Pictorial
Figure
s = 1000
representation of decoding representation
of the of dec
parity check? Hamming (7, 4) code
Hamming (7, 4) code. The
r1 r2 t
received vector is written = received
1000101
into the vector is wr
r3 The circles associated to diagram as shown in
diagram as shown in (a). In
t5 (b,c,d,e), the received vector(b,c,d,e),
is the receive
r7 r4 r n = 0000100
6 6 shown, assuming that the shown, assuming tha
(a) transmitted vector was as transmitted
in vector w
Which bits are involved figure 1.13b and therbits figure 1.13b and the
= 1000001
labelled
1* *
in all 10circles with a by ! were 1 flipped. The violated by ! were flipped. T
0 parity
* violation? parity checks are highlighted by checks are hig
1 0 dashed circles. One
11 0 1
1 0 0 1
dashed circles.
*
0 One of the seven
bits is the most prob
00 1 * 1
bits is the most probable suspect
only r5 ! (the flipped to bit) for each ‘syndrome’,
account to account for each ‘
0 11 0
0
0
0 1
1
0
0 00 1
i.e., 0
each pattern
0
of violated
i.e., each pattern of
and
(b) (c) (d) satisfied parity checks. satisfied parity check
(c) (d)
The syndrome to this In examples (b), (c),t and
5 = s In⊕examples
(d),
1 s
the 2 ⊕ s 3
(b), (c),
most probable suspe
most probable suspect is the one
error is based on the
bit that was flipped. 6
t = s ⊕
bit
2 that s3 ⊕wass4 flipped.
1 parity of the circles 1 In example (e), two bits have In example (e), two
1 1 t 7 = sflipped,
1 ⊕ s3 ⊕s3sand
been
4 t7 . T
flipped, s3 and t7 . The most
1 0 z = (100) 1 1 probable suspect is r
0 1 ! probable suspect is r2 , marked by
! 1*1 1* a circle in (e! ), which
1* 1* a circle in (e ), which shows the
!
output of the decodi
0* 0 0* output0 of the decoding algorithm.
0 00* 0 0 Course Overview - 58
0 0
(e) ! (e! )ht Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/05216429
nmitted. Printing
buy this not30
book for permitted.
pounds orhttp://www.cambridge.org/0521642981
$50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.
Decoding the Hamming
ce.phy.cam.ac.uk/mackay/itila/ for links.
Code (7, 4)
1.2: Error-correcting codes for the binary symmetric channel
mmetric channel 11
Figure 1.15. Pictoria
r5 Which circles violate the s = 1000
Figure 1.15. Pictorial representation of dec
parity check?
representation of decoding of the Hamming (7, 4) code
r1 r2 Hamming (7, 4) code. The received vector is wr
t = 1000101
r3 received Thevectorcircles associated
is written into the to diagram as shown in
diagram t5,ast6,shown
t7. in (a). In (b,c,d,e), the receive
r7 r4 r
6 (b,c,d,e), the received vector is n = 0010000
shown, assuming tha
(a) shown, assuming that the transmitted vector w
Which bits
transmitted vectorare wasinvolved
as in figure 1.13b and the
r = 1010101
1 figure
in all * and the
1.13b
0circles bitsalabelled 1
with by ! were flipped. T
1 byviolation?
! were flipped. The violated parity checks are hig
1 1* parity
1 checks 0are highlighted by 1 0 dashed circles. One
1 0 0 0
dashed circles. One of the seven 1* bits is the most prob
1* only
bits is the most r3 !probable
(the flipped
suspect bit) to account for each ‘
1 0 0 to1account 0 for each 0 ‘syndrome’, 1 0 i.e., each pattern of
0
0 (b) 1 0 (c) i.e., each pattern of violated (d) and satisfied parity check
0
(d) satisfied parity checks.to this
The syndrome t 5 = sIn
1 ⊕ s2 ⊕ s 3
examples (b), (c),
In examples (b), (c), and (d), the most probable suspe
error
most is based
probable suspect onisthe
the one t 6 = sbit
2 ⊕that
s3 ⊕was
s4 flipped.
1 bitparity
that was offlipped.
the circles 1 In example (e), two
In example (e), two bits have been
t 7 = sflipped,
1 ⊕ s3 ⊕ s3sand
4 t7 . T
1
1 0 flipped,zs= 3 and
(111) t7 . The
1 most 1 probable suspect is r
!
1 1* probable suspect is r2 , marked 1* by a circle in (e! ), which
1 output of the decodi
1* * a circle in (e !
), which* shows the
0 0 0 output of the decoding 0 0
algorithm.
0 Course Overview - 59
0* 0 0(e) (e! )0
(b) (c) (d) satis
In e
Decoding the Hamming Code (7, 4) mos
bit t
1 1 In e
! Flipping any one of the seven bits, we get a different syndrome is obtained in each flipp
1 0 1 1 prob
case. !
1* 1* a cir
outp
0*
! Seven non-zero syndromes, one
0 0 for each bit. 0* 0 0
! The all-zero(e)
syndrome (e! )
Alg
Syndrome z 000 001 010 011 100 101 110 111 the
Ham
Unflip this bit none r7 r6 r4 r5 r1 r2 r3 bin
sma
vec
che
If you try flipping any one of the seven bits, you’ll find that a different
! The optimal decoder unflips at most one bit, depending on the syndrome. (0),
syndrome is obtained in each case – seven non-zero syndromes, one for each the
bit.two-bit
! Any There is only
error one
pattern willother
lead tosyndrome, the all-zero
a decoded seven-bit vectorsyndrome. Sothree
that contains if the
channel is a binary symmetric channel with a small noise level f , the optimal
errors.
decoder unflips at most one bit, depending on the syndrome, as shown in
algorithm 1.16. Each syndrome could have been caused by other noise patterns
Course Overview - 60
too, but any other noise pattern that has the same syndrome must be lesss t s t s t Table 1.14. The sixteen codewords
{t} of the (7, 4) Hamming code.
Hamming
0100
0100110
0101
0101101
Code (7, 4):
1000 1000101
1001 1001110
Matricial
1100 1100011
1101 1101000
form Any pair of codewords differ from
each other in at least three bits.
0110 0110001 1010 1010010 1110 1110100
0111 0111010 1011 1011001 1111 1111111
! Because the Hamming code is a linear code, it can be written compactly in terms of
Hamming matrices
code is a linear code, it can be written compactly in
ces as follows. The transmittedCopyright
codewordCambridge
t is obtained from the Press 2003. On-screen viewing permitted. Printing not per
University
s and
ce s by a linear t as column vectors
operation, s and t as row vectors
You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mac
t = GTTs,
t=G s
(1.25) t = sG
10
e generator matrix of the code,
s1 s2 s3 s4
t1 1 0 0 0
t2 where t1 t2 t3 t4 t5 t6 t7
0 1 0 0 s1 1 0 0 0 1 0 1
t3 0 0 1 0
G G= =t4 0 0 0
T T
1 , (1.26) s2
0 1 0 0 1 1 0
.
t5
G = s3 0 0
G =
1 0 1 1 1
1 1 1 0
t6 0 1 1 1 s4 0 0 0 1 0 1 1
t7 1 0 1 1
I find it easier to relate to the right-multiplication (1.25) than the
t=5 0,=(1.27).
ing operation (1.25) uses modulo-2 arithmetic (1 + 1tion 1⊕
0s+ 1 =s2 ⊕ s3 coding theory texts use the left-multiplyin
Many
t = s ⊕ s however.
(1.27–1.28),
6 2 3 ⊕s 4
g operation (1.25) I have assumed that s and t are column vectors.
y are row vectors, then this equation is replaced by The
t =rows
s ⊕ofs the⊕ sgenerator matrix (1.28) can be viewed as defin
7 1
vectors lying3 in a4 seven-dimensional binary space. The sixteen
t = sG, (1.27)
obtained by making all possible linear combinations of these vec
Course Overview - 61
Decoding the (7, 4) Hamming codes t s t s t Table 1.14. The sixteen codewords
{t} of the (7, 4) Hamming code.
Decoding
0100
0101
0100110
0101101
the Hamming
1000 1000101
1001 1001110
Code (7,
1100 1100011
1101 1101000
4): Matricial view Any pair of codewords differ from
each other in at least three bits.
0110 0110001 1010 1010010 1110 1110100
0111 0111010 1011 1011001 1111 1111111
! Because the Hamming code is a linear code, it can be written compactly in terms of
Hamming matrices
code is a linear code, it can be written compactly in
ces as follows. The transmitted codeword t is obtained from the
ce s by a linears operation,
and t as column vectors
t = GTTs, (1.25)
t=G s
e generator matrix of the code,
s1 s2 s3 s4
t1 1 0 0 0
t2
0 1 0 0
t3 0 0 1 0
G G= =t4 0 0 0
T T
1 ,
(1.26)
t5 1 1 1 0
t6 0 1 1 1
t7 1 0 1 1
ing operation (1.25) uses modulo-2 arithmetic (1 + 1 = 0, 0 + 1 =
g operation (1.25) I have assumed that s and t are column vectors.
y are row vectors, then this equation is replaced by
t = sG, (1.27)
Course Overview - 62s t s t given
s by reading
t out itsTable
first 1.14. The sixteen
four bits. codewords is non-zero, then th
If the syndrome
noise sequence for this block{t} ofwas the non-zero,
(7, 4) Hamming
and thecode.
syndrome is our pointer t
Decoding the Hamming Code (7, 4): Matricial view
0100 0100110
0101 0101101
1000 1000101
1001 1001110
1100
the most1100011
probable errorAny
1101 1101000
pair of codewords differ from
pattern.
each other in at least three bits.
0110 0110001 1010 1010010 The
1110computation
1110100 of the syndrome vector is a linear operation. If we define th
0111 0111010 1011 1011001 31111
× 4 matrix
1111111 P such that the matrix of equation (1.26) is
! Because the Hamming code is a linear code, it can be written ! compactly
" in terms of
I4
GT = , (1.29
P
Hamming matrices
code is a linear code, it can be written compactly in
ces as follows. The transmitted codeword t where
is obtained
I4 is from
the 4the× 4 identity matrix, then the syndrome # vector
$ is z = Hr
ce s by a linears operation,
and t as column vectorswhere the parity-check matrix H is given by H = −P I3 ; in modulo
arithmetic, −1 ≡ 1, so
t = GTTs, (1.25)
t=G s
1 1 1 0 1 0 0
e generator matrix of the code,
H =H ⎡=
#
P P I I ⎤$
=
= 0 1 P 1 1 0 I13 0 . (1.30
s 1
t1 1 0 0 0
s 2 s 3 s 4 ⎣ 3 3
⎦ 1 0 1 1 0 0 1
t2
0 1I 0 0
t3 0 0 41 0 All the codewords t = G T
s of the code satisfy
The syndrome z is calculated with the
G G= =t4 0 0 0 1 ,
T T (1.26)
received sequence and H
t5 0
1 1 1 0
t6 0 1 P1 1 Ht = 0 . (1.31
t7 1 0 1 1 z = Hr 0
ing operation (1.25) uses modulo-2 arithmetic (1 + 1 =
! Exercise 0,[10 ]+Prove
1.4. 1 = that this is so by evaluating the 3 × 4 matrix HGT.
All the codewords t of the code satisfy
g operation (1.25) I have assumed that s and Since ⎡vectors.
0 ⎤vector r is given by r = GTs + n, the syndrome-decodin
the received
t are column
problem
y are row vectors, then this equation is replaced by is to⎢ find ⎥ = Ht
0 the most probable noise vector n satisfying the equation
⎢ ⎥
t = sG, ⎢⎣ 0 ⎥
(1.27)
⎦ Hn = z. (1.32
A decoding algorithm that solves this problem is called a maximum-likelihoo
Course Overview - 63
decoder. We will discuss decoding problems like this in later chapters.You can also read
