￼Video Flashing Reduction… Video Flashing Reduction Background and Algorithm March 2023 v1.0

Important: This document includes a sample video that contains sequences of flashing effects. Important: This document includes a sample video that contains sequences of flashing effects.Contents Contents￼

Introduction

Background

General Description of the Algorithm

Universal Flicker Metric

Technical Challenges

Detailed Description of the Algorithm

Algorithm Architecture

Static Inputs

Unit Luminance

Standard Sizes

Standard Luminances

Standard Frame Rates

Video Area and Size

Filter Selection

Frame Processing

Normalized Average Luminance

Adapting Normalized Luminance

Luminance Contrast

Filter Responses

Energy Pooling

Energy Interpolation

Energy Adjustment

Risk Mapping

Step Masking

Mitigation

Mitigation Transfer Function

Mitigation Strength

Mitigation Fading

Mitigation Factors

Example: High Contrast Pulse Train

Example: Detection and Mitigation of Video Flashing

Future Work

Appendices

Appendix: Contrast Filters

Appendix: Notation

Bibliography

Rectangle RectangleIntroduction

Background

Video content with bright, repetitive, high-contrast flashing may be uncomfortable for some viewers. In addition, more serious conditions, such as photosensitivity and photosensitive epilepsy, may be triggered by bright, flashing video content. In this paper, we describe an algorithm to detect and mitigate potentially uncomfortable video flashing.

Data on the precise stimulus attributes that produce discomfort in viewers are few and fragmentary. One widely cited report describes the fraction of symptomatic patients who experience photo-paroxysmal response (PPR) EEG waveforms as a function of the frequency of a flashing stimulus (G. F. Harding & Jeavons, 1994).

PPR are considered a precursor of epileptic seizures. A plot of those data are shown in Figure 1. They show that the incidence is highest between 10 and 50 Hz. Other data show that these responses increase with area, contrast, and luminance (G. Harding & Fylan, 1999; Trenité, 2006; Wilkins et al., 1980).

￼

Not all discomfort is confined to clinical conditions. In a recent study, observers rated their discomfort with sinusoidally flashing uniform fields of various frequencies (Gentile & Aguirre, 2020). The contrast was approximately 0.9, and the luminance about 150 nits. Results are shown in Figure 2. The points show that discomfort (on a 10-point scale) was greatest for frequencies between 8 and 30 Hz. The blue curve shows the sensitivity of the Universal Flicker Metric (UFM), a metric we developed to predict sensitivity to temporal luminance fluctuations at arbitrary display sizes and luminances. This rough agreement, and the fact that the UFM can adapt to size and luminance, encouraged us to use it as the basis of our video flashing metric. The UFM is discussed further below.

Rectangle Rectangle￼

General Description of the Algorithm

The algorithm processes a stream of video frames, and identifies those frames or groups of frames that are likely to cause discomfort. The algorithm achieves this outcome by first computing the average luminance of each frame, and then applying filters to the sequence of frames to select energy at frequencies that are known to present a risk of discomfort. We then accumulate the energy over a brief interval of time, and convert that accumulated energy into a risk of discomfort, using a risk-mapping function.

In addition, for users that opt in to this feature, the algorithm actively reduces the risk through a mitigation process that primarily reduces the luminance and contrast of the video during the identified segments.

Universal Flicker Metric

The Universal Flicker Metric is a tool for evaluating the visibility of temporal modulations of luminance (Watson & Agaoglu, 2018). It is generally useful as a tool for evaluating flicker in displays, but also serves as a general model of human sensitivity to variations in luminance over time. An example of the predictions of the metric for sinusoidal variations is shown in Figure 2 above. In general, the UFM accepts as input a record of luminance over time, as well as an indication of the illuminated area, and outputs a measure of the visibility of fluctuations from the mean in units of Just Noticeable Difference (JND).

Technical Challenges

Because the algorithm is designed to operate in real time with minimal delay, all operations are causal and work only with the current frame and past results. As a consequence, it isn’t possible to mitigate the potentially risky frames before they are detected. Thus, in a sequence of flashes, the first few may be unmitigated. Within the constraints of an effective method, we have tried to keep delays to a minimum.

Because visual sensitivity to flicker varies substantially with adapting luminance and flickering area, it is necessary to adopt filters that reflect the current size and luminance. However the adapting luminance, at least, can vary by a large factor over the course of a video. Our solution to this challenge is to select a set of five filters, suitable for a large range of luminance, and interpolate from the energy of their outputs based on the current adapting luminance.

Detailed Description of the Algorithm

Algorithm Architecture

Video Flashing Reduction (VFR) includes both a Video Flashing Metric (VFM) to quantify the risk of discomfort, and an algorithm for mitigating the risk. VFM is designed to accommodate either real-time streaming video or off-line, non-real-time video processing that can be used to annotate a video file. An overview of the VFR architecture is shown in Figure 3. The algorithm has two types of input: static, which are provided once per video or session, and dynamic, which consist of the stream of frames.

￼

Static Inputs

There are three static, one-time or per-session inputs: U, the unit luminance of the display in nits, A, the area of the video in square degrees, and W₀, the frame rate of the video in Hz.

Unit Luminance

The unit luminance U is equivalent to the peak white luminance, produced when the normalized luminance is 1. Normalized luminance is the luminance of a pixel divided by the peak white luminance.

Standard Sizes

The VFM employs a set of three standard sizes: 6, 20, and 45 degrees, corresponding roughly to the longer dimensions of Apple Watch, iPhone, and MacBook displays. These values are derived from the UFM.

Standard Luminances

Likewise VFM employs five standard luminances: 0.2, 1, 10, 150, and 500 nits (cd m⁻²). These values are also derived from the UFM.

Standard Frame Rates

The VFM handles a set of 7 standard frame rates: 24, 25, 30, 50, 60, 90, 120 Hz. If the input frame rate W₀ differs from these rates, we select the nearest standard frame rate W. We call W the video frame rate.

Video Area and Size

The area A in square degrees depends on the area of the display in square meters Am, and the viewing distance in meters d:

￼

(1)

The video area A is converted to a size S₀, based on the formula:

￼

(2)

We then find the nearest standard size S to S₀. We call S the video size.

Filter Selection

The VFM is provided with a set of 105 digital FIR filter kernels. These correspond to a separate kernel for each standard frame rate, standard size, and standard luminance. These kernels are based on the UFM. Consult the Appendix for more details on the construction of these filters.

At the start of a session, a set of five kernels is selected, written hk, corresponding to the video frame rate W and size S, as defined above. The subscript k is an index to the five standard luminances. An example set of five kernels is shown in Figure 4. The kernels may have different lengths, which we write Nk.

￼Group pasted-image.pdf Caption: Figure 4. Five filter kernels for W= 24 Hz, S = 20 deg. Figure 4. Five filter kernels for W= 24 Hz, S = 20 deg.

Frame Processing

The remaining inputs are the video frames. Frame processing is a continuous streaming process, and in general we refer to frame number i as the current frame. A number of process steps involve previous steps. The process of including previous steps is handled by means of buffers of appropriate length, that are generally initialized to zero.

Normalized Average Luminance

The first step in the process is to convert each video frame to a normalized average luminance l(i), where i is the frame number. Normalized means that the maximum numerical luminance value in the video container is 1. Conversion to luminance depends on the specific encoding of the digital video.

Adapting Normalized Luminance

The adapting normalized luminance a(i) is computed by a recursive IIR (infinite impulse response) exponential filter, based on l(i) and a(i-1):

￼ .

(3)

The rate of change of the adaptation level is determined by the parameter 𝜇, which depends on the parameter 𝜏 (in seconds) and the frame rate W:

￼

(4)

The adaptation level must be initialized at i = 0 (the frame before the first), and we set a(0) = l(1).

Luminance Contrast

The contrast c(i) is computed as:

￼ .

(5)

Filter Responses

The contrast waveform c(i) is then convolved with the five selected kernels hk to produce the five responses rk(i):

￼ .

(6)

Energy Pooling

At the next stage, we compute the local energy of the each response. The response rk(i) is raised to the power 𝛼, convolved with an energy kernel he and the 𝛼 root taken to compute the local energy ek(i):

￼ .

(7)

Typically the exponent 𝛼 = 2. Note that we call this energy, but it is actually the square root of energy. Ne is the length of the energy kernel.

The energy pooling kernel he is defined as the PDF of a Gamma probability density function with shape parameter 𝜅 and scale parameter 𝜆 (seconds):

￼

(8)

The sampled discrete version of the kernel must be constructed based on the frame rate W. An example is pictured in Figure 5.

￼Group pasted-image.pdf Caption: Figure 5. Energy pooling kernel for W = 24 Hz, 𝜅 = 2, 𝜆 = 0.15. Figure 5. Energy pooling kernel for W = 24 Hz, 𝜅 = 2, 𝜆 = 0.15.

Energy Interpolation

We determine the final energy e(i) by interpolating among the five energies ek(i), based on the value of the adapting normalized luminance a(i) times the unit luminance U, relative to the standard luminances of the kernels. The interpolation is linear in the space of energy versus log adaptation level, as shown in Figure 6. This interpolation solves the problem of how to adapt the filter kernel to the current adaption luminance.

￼

Energy Adjustment

Following interpolation, we multiply by a term g, an adjustment factor determined by the input size S₀ and a global gain G:

￼

(9)

This term adjusts for the difference between the actual size and the standard size, and adjusts for different frame rates in subsequent energy calculations. The parameter G is a global gain parameter, and 𝛼 is the energy pooling exponent described in the previous section.

Risk Mapping

We can map the energy to a bounded quantity that reflects the risk of discomfort. The energy computed above is bounded, in practical terms and using default parameters, to about 4,000. But most video produces values of less than 500. We have adopted a particular mapping that suppresses small values, and compresses large values. The mapping is based on a Weibull probability density, with location parameter 𝜇, shape parameter 𝜈, and scale parameter 𝜉. The mapping to risk q can be written:

￼

(10)

This mapping is illustrated in Figure 7 for the default parameter values.

￼

Step Masking

The algorithm developed above correctly estimates risk as the supra-threshold magnitude of the contrast energy. But it may respond to transients that, while highly visible, are less likely to lead to discomfort. In particular, scene cuts may involve large, abrupt step changes in mean luminance. To ameliorate this effect, we include an additional process that we call step masking.

Step masking involves a parallel process, similar to the energy calculations above, but differing in that normalized cross-correlation is used in place of convolution.

Outside of the frame processing loop, after kernels are selected, we compute the norm of each kernel:￼

￼ .

(11)

Within the frame processing loop, we compute the norm of the luminance contrast over the preceding Nk frames:

￼ .

(12)

We then cross-correlate the contrast waveform c(i) with the five selected kernels hk to produce the five normalized cross-correlations nk(i):

￼ .

(13)

Note that this expression is similar to the convolutions described by Equation 6, but the kernel isn’t reversed in time before multiplication, and the result is normalized by the kernel and contrast norms.

The energies of these correlations are then pooled, as with the energies above, through convolution with the energy kernel:

￼ .

(14)

We again interpolate among the five correlation energies, to yield a final correlation energy f(i). Finally, if this quantity for a frame is below a threshold 𝛿, then the risk is masked to zero.

Mitigation

Mitigation Transfer Function

To reduce the risk of discomfort posed by flashing video, we use a simple reduction of contrast. Specifically, we use a transfer function described by:

￼

(15)

where lin and lout are the input and output luminances, and fC and fL are the contrast and luminance mitigation factors, and a is the adapting luminance for that frame. Examples of this transfer function are shown in Figure 8.

￼

Currently the mitigation is applied on a frame-by-frame basis. Note that the transfer operates on linear normalized luminance values, and for an RGB color image acts equally and independently on the three color channels. An illustration of the effects of fC and fL on a single frame is shown in Figure 9.

￼

In Figure 10 we show an animation in which the contrast and luminance mitigation factors are separately varied between 0 and 1.

￼Figure 10. Animation showing the effect of contrast and luminance mitigation. Figure 10. Animation showing the effect of contrast and luminance mitigation.

Mitigation Strength

The mitigation strength is given by:

￼

(16)

where M is the mitigation gain in the range [0,1], and q(i) is the frame risk. Because risk is in the range [0,100], the log is in the range [0,2], and mitigation strength is in the range [0,1].

Mitigation Fading

Additionally, to provide a gentle recovery from mitigation, when mitigation declines we adjust m with an exponential decay over time, using a timescale (in units of frame) of 𝜇m:

￼ .

(17)

The rate of change of the mitigation strength is determined by the parameter 𝜇m, which depends on the parameter 𝜏m (in seconds) and the frame rate W:

￼

(18)

The mitigation strength must be initialized at i = 0 (the frame before the first), and we set m(0) = 0.

Mitigation Factors

We have set the mitigation factors for contrast and luminance in each frame according to the formula:

{fC , fL }= 1 - m {wC , wL }

(19)

where m is the mitigation strength, fC and fL are the contrast and luminance mitigation factors, and wC , wL are the contrast and luminance mitigation weights. The weights are in the range [0,1], and so the factors will lie in the range [0,1].

Example: High Contrast Pulse Train

In this section we illustrate the steps in the algorithm with a synthetic sequence. While it is unlikely to occur in typical video, a frame sequence that is particularly risky is an an extended sequence of alternating bright and dark frames. The risk increases when the alternation rate is near to the peak of visual sensitivity (see Figure 2). In this example, we use a frame rate of 24 Hz and alternating frames of bright (normalized luminance = 1) and dim (0.2), yielding a signal at 12 Hz. The minimum luminance is 100 nits, and the maximum 500 nits. There are a total of 36 pulses, over an interval of 3 seconds.

The processing steps are illustrated in Figure 11. The adaptation level (panel 2) begins at the lower luminance but rises as the flashes begin, and subsides once they cease. The contrast (panel 3) is initially very high, but subsides somewhat as adaptation level rises. The responses (panel 4) are also very large for the first pulses, but decline somewhat. The energy (panel 5) rises to a level of around 500, and remains high until the pulses cease. The risk (panel 6) rises quickly to the maximum of 100 and remains there until after the pulses cease. In panel 7, we show the mitigated luminance after the first few pulses the contrast is greatly attenuated. In panels 8 and 9, we show the mitigated energy and risk.

￼Group pasted-image.pdf Caption: Figure 11. Example of processing stages for input of high contrast pulse train. Figure 11. Example of processing stages for input of high contrast pulse train.

Example: Detection and Mitigation of Video Flashing

Here we show an example of a video clip with significant video flashing. We show both original and mitigated video, along with a plot of the energy and mitigation strength over the time course of the video.

￼VideoFlashingReduction-Fig12.mp4

Future Work

As we develop a deeper understanding of the specific video features that induce discomfort, and how they can be mitigated, and as we gain more experience with our current algorithm, we anticipate future enhancements that will be made available as part of the usual software upgrade process. We expect those enhancements will also be documented.

Appendices

Appendix: Contrast Filters

The metric relies on contrast filters that process the stream of frame luminance values. Each filter is an finite impulse response (FIR) defined by a kernel (a list of numbers). There is a distinct kernel for each combination of average luminance, display size, and frame rate. In this section, we describe the creation of these filters.

The filters are based on the filters defined by the Universal Flicker Metric. These filters were created by fitting a particular function to the contrast sensitivity data collected in a user study (Watson & Agaoglu, 2018). That function represents a particular filter design, and is specified by five parameters: {g, 𝛼, 𝛽, a, b, c}. The filter can be expressed as either an impulse response h, or a transfer function H. Both are continuous functions.

Note that video at a frame rate W will have temporal frequencies up to W/2. If the kernel has an odd number N of samples, then its total duration will be D = N/W. The MTF=|DFT| of that kernel will specify magnitudes at a discrete set of frequencies with spacing dW = 1/D = W/N, from 0 to dW (N-1)/2 = (W/2)(N-1)/2. We call that mtf₀.

One approach to creating an FIR kernel would be to sample the impulse response at discrete points in time that are multiples of the frame rate. We call that ker₁. However, the mtf₁ of ker₁ may not resemble mtf₀. This typically happens when the frame rate is low, because the samples are too far apart in time.

However, we can perturb the filter parameters to produce a ker₂ whose mtf₂ resembles mtf₀. We have found that this generally produces a good match between mtf₂ and mtf₀, so long as certain conditions are met. Figure 13 shows this process for one example at 24 Hz frame rate.

￼
￼

Figure 13. Construction of UFM filter kernels. The individual plots are, on the left, ker1 (red): the initial estimate of the kernel; ker2 (green): final estimate of the kernel; on the right, mtf1 (red) the initial MTF; mtf1: the final MTF; mtf0 (blue): the MTF of the UFM.

In the UFM, we defined filter parameters for three sizes: 6, 20, and 45 degrees, and for 5 mean luminances: 0.2, 1, 10, 150, and 500 nits. (The sizes refer to the longer dimension of a rectangle whose other dimensions was 1.6 times shorter.) Here we create filters for that same array of nits and size, and also for a set of seven possible frame rates: 24, 25, 30, 50, 60, 90, and 120 Hz. This results in a total number of kernels of 3 x 5 x 7 = 105 kernels.

The required kernel length is determined by the frame rate, and by the duration of the impulse response, which increases as luminance is reduced and as size is reduced. The longest kernel is required for the frame rate of luminance of 0.2 nits, and size of 6 degrees, of about 250 ms. At 120 Hz, that results in a kernel of length 31. Table 1 below shows the required kernel length when terminal values less than 10⁻⁴ are dropped.

￼Group pasted-image.pdf Caption: Table 1. Kernel lengths. Table 1. Kernel lengths.

The complete set of kernels is shown in Figure 14. The MTFs of these kernels are shown in Figure 15, along with the target MTF derived from the transfer function of the UFM.

￼

￼

Appendix: Notation

The following is a summary of notation used in this document, in order of appearance. Where relevant default values and units are provided.

Symbol
Default
Definition
Unit

U
500
Peak display luminance
nits

A
1265.63
Display area
degree²

W

Frame rate of selected kernel
Hz

W₀

Input frame rate
Hz

Am

Display area
meters²

d

Display viewing distance
meter

S₀

Input display size
degree

S

Kernel display size
degree

hk

Kernel for kth standard luminance

Nk

Length of kth kernel

i

Frame number

l(i)

Frame normalized average luminance

a(i)

Frame adapting normalized luminance

𝜇L

Adapting luminance filter coefficient

𝜏L

Adapting luminance time constant
seconds

c(i)

Frame contrast

rk(i)

Frame response of kth filter

he

Energy kernel

𝛼
2
Energy exponent

𝜆
0.15
Scale of energy kernel
seconds

𝜅
2
Shape of energy kernel

ek(i)

Frame energy of kth kernel

e(i)

Frame energy

g

Adjustment factor

G
1
Global gain parameter

cA
0.263
Area effect on sensitivity

q(i)

Frame risk

𝜈
3
Risk map shape parameter

𝜉
200
Risk map scale parameter

𝜇
33
Risk map threshold parameter

nk(i)

Normalized cross-correlation for kernel k

fk(i)

Cross-correlation energy for kernel k

f(i)

Cross-correlation energy

𝛿
1.8
Cross-correlation energy threshold

fC

Contrast mitigation factor

fL

Luminance mitigation factor

m

Mitigation strength

M
1
Mitigation gain

𝜇m

Mitigation filter coefficient

𝜏m
2
Mitigation time constant
seconds

wC
1
Contrast mitigation weight

wL
0.5
Luminance mitigation weight

Bibliography

Gentile, C. P., & Aguirre, G. K. (2020). A neural correlate of visual discomfort from flicker. Journal of Vision, 20(7), 11-11.

Harding, G., & Fylan, F. (1999). Two visual mechanisms of photosensitivity. Epilepsia, 40(10), 1446-1451.

Harding, G. F., & Jeavons, P. M. (1994). Photosensitive epilepsy: Cambridge University Press.

Trenité, D. G. K.-N. (2006). Photosensitivity, visually sensitive seizures and epilepsies. Epilepsy research, 70, 269-279.

Watson, A. B., & Agaoglu, M. (2018). Universal Flicker Metric: Apple.

Wilkins, A., Binnie, C., & Darby, C. (1980). Visually-induced seizures. Progress in neurobiology, 15(2), 85-117.