webrtc_m130/video/buffered_frame_decryptor.h

106 lines
4.4 KiB
C
Raw Normal View History

Add BufferedFrameDecryptor to cleanly deal with receiving encrypted frames. This change introduces a new class BufferedFrameDecryptor that is responsible for decrypting received encrypted frames and passing them on to the RtpReferenceFinder. This decoupling refactoring was triggered by a new optimization also introduced in this patch to stash a small number of undecryptable frames if no frames have ever been decrypted. The goal of this optimization is to prevent re-fectching of key frames on low bandwidth networks simply because the key to decrypt them had not arrived yet. The optimization will stash 24 frames (about 1 second of video) in a ring buffer and will attempt to re-decrypt previously received frames on the first valid decryption. This allows the decoder to receive the key frame without having to request due to short key delivery latencies. In testing this is actually hit quite often and saves an entire RTT which can be up to 200ms on a bad network. As the scope of frame encryption increases in WebRTC and has more specialized optimizations that do not apply to the general flow it makes sense to move it to a more explicit bump in the stack protocol that is decoupled from the WebRTC main flow, similar to how SRTP is utilized with srtp_protect and srtp_unprotect. One advantage of this approach is the BufferedFrameDecryptor isn't even constructed if FrameEncryption is not in use. I have decided against merging the RtpReferenceFinder and EncryptedFrame stash because it introduced a lot of complexity around the mixed scenario where some of the frames in the stash are encrypted and others are not. In this case we would need to mark certain frames as decrypted which appeared to introduce more complexity than this simple decoupling. Bug: webrtc:10022 Change-Id: Iab74f7b7d25ef1cdd15c4a76b5daae1cfa24932c Reviewed-on: https://webrtc-review.googlesource.com/c/112221 Commit-Queue: Benjamin Wright <benwright@webrtc.org> Reviewed-by: Philip Eliasson <philipel@webrtc.org> Reviewed-by: Stefan Holmer <stefan@webrtc.org> Cr-Commit-Position: refs/heads/master@{#25865}
2018-11-30 16:18:26 -08:00
/*
* Copyright (c) 2018 The WebRTC project authors. All Rights Reserved.
*
* Use of this source code is governed by a BSD-style license
* that can be found in the LICENSE file in the root of the source
* tree. An additional intellectual property rights grant can be found
* in the file PATENTS. All contributing project authors may
* be found in the AUTHORS file in the root of the source tree.
*/
#ifndef VIDEO_BUFFERED_FRAME_DECRYPTOR_H_
#define VIDEO_BUFFERED_FRAME_DECRYPTOR_H_
#include <deque>
#include <memory>
#include "api/crypto/crypto_options.h"
#include "api/crypto/frame_decryptor_interface.h"
#include "api/field_trials_view.h"
Add BufferedFrameDecryptor to cleanly deal with receiving encrypted frames. This change introduces a new class BufferedFrameDecryptor that is responsible for decrypting received encrypted frames and passing them on to the RtpReferenceFinder. This decoupling refactoring was triggered by a new optimization also introduced in this patch to stash a small number of undecryptable frames if no frames have ever been decrypted. The goal of this optimization is to prevent re-fectching of key frames on low bandwidth networks simply because the key to decrypt them had not arrived yet. The optimization will stash 24 frames (about 1 second of video) in a ring buffer and will attempt to re-decrypt previously received frames on the first valid decryption. This allows the decoder to receive the key frame without having to request due to short key delivery latencies. In testing this is actually hit quite often and saves an entire RTT which can be up to 200ms on a bad network. As the scope of frame encryption increases in WebRTC and has more specialized optimizations that do not apply to the general flow it makes sense to move it to a more explicit bump in the stack protocol that is decoupled from the WebRTC main flow, similar to how SRTP is utilized with srtp_protect and srtp_unprotect. One advantage of this approach is the BufferedFrameDecryptor isn't even constructed if FrameEncryption is not in use. I have decided against merging the RtpReferenceFinder and EncryptedFrame stash because it introduced a lot of complexity around the mixed scenario where some of the frames in the stash are encrypted and others are not. In this case we would need to mark certain frames as decrypted which appeared to introduce more complexity than this simple decoupling. Bug: webrtc:10022 Change-Id: Iab74f7b7d25ef1cdd15c4a76b5daae1cfa24932c Reviewed-on: https://webrtc-review.googlesource.com/c/112221 Commit-Queue: Benjamin Wright <benwright@webrtc.org> Reviewed-by: Philip Eliasson <philipel@webrtc.org> Reviewed-by: Stefan Holmer <stefan@webrtc.org> Cr-Commit-Position: refs/heads/master@{#25865}
2018-11-30 16:18:26 -08:00
#include "modules/video_coding/frame_object.h"
namespace webrtc {
// This callback is provided during the construction of the
// BufferedFrameDecryptor and is called each time a frame is sucessfully
// decrypted by the buffer.
class OnDecryptedFrameCallback {
public:
virtual ~OnDecryptedFrameCallback() = default;
// Called each time a decrypted frame is returned.
virtual void OnDecryptedFrame(std::unique_ptr<RtpFrameObject> frame) = 0;
Add BufferedFrameDecryptor to cleanly deal with receiving encrypted frames. This change introduces a new class BufferedFrameDecryptor that is responsible for decrypting received encrypted frames and passing them on to the RtpReferenceFinder. This decoupling refactoring was triggered by a new optimization also introduced in this patch to stash a small number of undecryptable frames if no frames have ever been decrypted. The goal of this optimization is to prevent re-fectching of key frames on low bandwidth networks simply because the key to decrypt them had not arrived yet. The optimization will stash 24 frames (about 1 second of video) in a ring buffer and will attempt to re-decrypt previously received frames on the first valid decryption. This allows the decoder to receive the key frame without having to request due to short key delivery latencies. In testing this is actually hit quite often and saves an entire RTT which can be up to 200ms on a bad network. As the scope of frame encryption increases in WebRTC and has more specialized optimizations that do not apply to the general flow it makes sense to move it to a more explicit bump in the stack protocol that is decoupled from the WebRTC main flow, similar to how SRTP is utilized with srtp_protect and srtp_unprotect. One advantage of this approach is the BufferedFrameDecryptor isn't even constructed if FrameEncryption is not in use. I have decided against merging the RtpReferenceFinder and EncryptedFrame stash because it introduced a lot of complexity around the mixed scenario where some of the frames in the stash are encrypted and others are not. In this case we would need to mark certain frames as decrypted which appeared to introduce more complexity than this simple decoupling. Bug: webrtc:10022 Change-Id: Iab74f7b7d25ef1cdd15c4a76b5daae1cfa24932c Reviewed-on: https://webrtc-review.googlesource.com/c/112221 Commit-Queue: Benjamin Wright <benwright@webrtc.org> Reviewed-by: Philip Eliasson <philipel@webrtc.org> Reviewed-by: Stefan Holmer <stefan@webrtc.org> Cr-Commit-Position: refs/heads/master@{#25865}
2018-11-30 16:18:26 -08:00
};
// This callback is called each time there is a status change in the decryption
// stream. For example going from a none state to a first decryption or going
// frome a decryptable state to a non decryptable state.
class OnDecryptionStatusChangeCallback {
public:
virtual ~OnDecryptionStatusChangeCallback() = default;
// Called each time the decryption stream status changes. This call is
// blocking so the caller must relinquish the callback quickly. This status
// must match what is specified in the FrameDecryptorInterface file. Notably
// 0 must indicate success and any positive integer is a failure.
Reland "Refactor FrameDecryptorInterface::Decrypt to use new API." This reverts commit 7dd83e2bf73a7f1746c5ee976939bf52e19fa8be. Reason for revert: This wasn't the cause of the break. Original change's description: > Revert "Refactor FrameDecryptorInterface::Decrypt to use new API." > > This reverts commit 642aa81f7d5cc55d5b99e2abc51327eed9d40195. > > Reason for revert: Speculative revert. The chromium roll is failing: > https://ci.chromium.org/p/chromium/builders/try/linux-rel/64388 > But I can't figure out exactly what is failing, this looks suspecious. > > Original change's description: > > Refactor FrameDecryptorInterface::Decrypt to use new API. > > > > This change refactors the FrameDecryptorInterface to use the new API. The new > > API surface simply moves bytes_written to the return type and implements a > > simple Status type. > > > > Bug: webrtc:10512 > > Change-Id: I622c5d344d58e618853c94c2f691cf7c8fb73a36 > > Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/131460 > > Reviewed-by: Steve Anton <steveanton@webrtc.org> > > Reviewed-by: Fredrik Solenberg <solenberg@webrtc.org> > > Reviewed-by: Rasmus Brandt <brandtr@webrtc.org> > > Reviewed-by: Stefan Holmer <stefan@webrtc.org> > > Commit-Queue: Benjamin Wright <benwright@webrtc.org> > > Cr-Commit-Position: refs/heads/master@{#27497} > > TBR=brandtr@webrtc.org,steveanton@webrtc.org,solenberg@webrtc.org,ossu@webrtc.org,stefan@webrtc.org,benwright@webrtc.org > > Change-Id: Ia9ec70263762c34671af13f0d519e636eb8473cd > No-Presubmit: true > No-Tree-Checks: true > No-Try: true > Bug: webrtc:10512 > Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/132013 > Reviewed-by: Henrik Boström <hbos@webrtc.org> > Commit-Queue: Henrik Boström <hbos@webrtc.org> > Cr-Commit-Position: refs/heads/master@{#27510} TBR=brandtr@webrtc.org,steveanton@webrtc.org,solenberg@webrtc.org,hbos@webrtc.org,ossu@webrtc.org,stefan@webrtc.org,benwright@webrtc.org Change-Id: I8e4b7965cf1d1a1554c3b46e6245f5ad0d2dcbb4 No-Presubmit: true No-Tree-Checks: true No-Try: true Bug: webrtc:10512 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/131982 Reviewed-by: Benjamin Wright <benwright@webrtc.org> Commit-Queue: Benjamin Wright <benwright@webrtc.org> Cr-Commit-Position: refs/heads/master@{#27529}
2019-04-09 20:08:41 +00:00
virtual void OnDecryptionStatusChange(
FrameDecryptorInterface::Status status) = 0;
};
Add BufferedFrameDecryptor to cleanly deal with receiving encrypted frames. This change introduces a new class BufferedFrameDecryptor that is responsible for decrypting received encrypted frames and passing them on to the RtpReferenceFinder. This decoupling refactoring was triggered by a new optimization also introduced in this patch to stash a small number of undecryptable frames if no frames have ever been decrypted. The goal of this optimization is to prevent re-fectching of key frames on low bandwidth networks simply because the key to decrypt them had not arrived yet. The optimization will stash 24 frames (about 1 second of video) in a ring buffer and will attempt to re-decrypt previously received frames on the first valid decryption. This allows the decoder to receive the key frame without having to request due to short key delivery latencies. In testing this is actually hit quite often and saves an entire RTT which can be up to 200ms on a bad network. As the scope of frame encryption increases in WebRTC and has more specialized optimizations that do not apply to the general flow it makes sense to move it to a more explicit bump in the stack protocol that is decoupled from the WebRTC main flow, similar to how SRTP is utilized with srtp_protect and srtp_unprotect. One advantage of this approach is the BufferedFrameDecryptor isn't even constructed if FrameEncryption is not in use. I have decided against merging the RtpReferenceFinder and EncryptedFrame stash because it introduced a lot of complexity around the mixed scenario where some of the frames in the stash are encrypted and others are not. In this case we would need to mark certain frames as decrypted which appeared to introduce more complexity than this simple decoupling. Bug: webrtc:10022 Change-Id: Iab74f7b7d25ef1cdd15c4a76b5daae1cfa24932c Reviewed-on: https://webrtc-review.googlesource.com/c/112221 Commit-Queue: Benjamin Wright <benwright@webrtc.org> Reviewed-by: Philip Eliasson <philipel@webrtc.org> Reviewed-by: Stefan Holmer <stefan@webrtc.org> Cr-Commit-Position: refs/heads/master@{#25865}
2018-11-30 16:18:26 -08:00
// The BufferedFrameDecryptor is responsible for deciding when to pass
// decrypted received frames onto the OnDecryptedFrameCallback. Frames can be
// delayed when frame encryption is enabled but the key hasn't arrived yet. In
// this case we stash about 1 second of encrypted frames instead of dropping
// them to prevent re-requesting the key frame. This optimization is
// particularly important on low bandwidth networks. Note stashing is only ever
// done if we have never sucessfully decrypted a frame before. After the first
// successful decryption payloads will never be stashed.
class BufferedFrameDecryptor final {
public:
// Constructs a new BufferedFrameDecryptor that can hold
explicit BufferedFrameDecryptor(
OnDecryptedFrameCallback* decrypted_frame_callback,
OnDecryptionStatusChangeCallback* decryption_status_change_callback,
const FieldTrialsView& field_trials);
Add BufferedFrameDecryptor to cleanly deal with receiving encrypted frames. This change introduces a new class BufferedFrameDecryptor that is responsible for decrypting received encrypted frames and passing them on to the RtpReferenceFinder. This decoupling refactoring was triggered by a new optimization also introduced in this patch to stash a small number of undecryptable frames if no frames have ever been decrypted. The goal of this optimization is to prevent re-fectching of key frames on low bandwidth networks simply because the key to decrypt them had not arrived yet. The optimization will stash 24 frames (about 1 second of video) in a ring buffer and will attempt to re-decrypt previously received frames on the first valid decryption. This allows the decoder to receive the key frame without having to request due to short key delivery latencies. In testing this is actually hit quite often and saves an entire RTT which can be up to 200ms on a bad network. As the scope of frame encryption increases in WebRTC and has more specialized optimizations that do not apply to the general flow it makes sense to move it to a more explicit bump in the stack protocol that is decoupled from the WebRTC main flow, similar to how SRTP is utilized with srtp_protect and srtp_unprotect. One advantage of this approach is the BufferedFrameDecryptor isn't even constructed if FrameEncryption is not in use. I have decided against merging the RtpReferenceFinder and EncryptedFrame stash because it introduced a lot of complexity around the mixed scenario where some of the frames in the stash are encrypted and others are not. In this case we would need to mark certain frames as decrypted which appeared to introduce more complexity than this simple decoupling. Bug: webrtc:10022 Change-Id: Iab74f7b7d25ef1cdd15c4a76b5daae1cfa24932c Reviewed-on: https://webrtc-review.googlesource.com/c/112221 Commit-Queue: Benjamin Wright <benwright@webrtc.org> Reviewed-by: Philip Eliasson <philipel@webrtc.org> Reviewed-by: Stefan Holmer <stefan@webrtc.org> Cr-Commit-Position: refs/heads/master@{#25865}
2018-11-30 16:18:26 -08:00
~BufferedFrameDecryptor();
// This object cannot be copied.
BufferedFrameDecryptor(const BufferedFrameDecryptor&) = delete;
BufferedFrameDecryptor& operator=(const BufferedFrameDecryptor&) = delete;
// Sets a new frame decryptor as the decryptor for the buffered frame
// decryptor. This allows the decryptor to be switched out without resetting
// the video stream.
void SetFrameDecryptor(
rtc::scoped_refptr<FrameDecryptorInterface> frame_decryptor);
Add BufferedFrameDecryptor to cleanly deal with receiving encrypted frames. This change introduces a new class BufferedFrameDecryptor that is responsible for decrypting received encrypted frames and passing them on to the RtpReferenceFinder. This decoupling refactoring was triggered by a new optimization also introduced in this patch to stash a small number of undecryptable frames if no frames have ever been decrypted. The goal of this optimization is to prevent re-fectching of key frames on low bandwidth networks simply because the key to decrypt them had not arrived yet. The optimization will stash 24 frames (about 1 second of video) in a ring buffer and will attempt to re-decrypt previously received frames on the first valid decryption. This allows the decoder to receive the key frame without having to request due to short key delivery latencies. In testing this is actually hit quite often and saves an entire RTT which can be up to 200ms on a bad network. As the scope of frame encryption increases in WebRTC and has more specialized optimizations that do not apply to the general flow it makes sense to move it to a more explicit bump in the stack protocol that is decoupled from the WebRTC main flow, similar to how SRTP is utilized with srtp_protect and srtp_unprotect. One advantage of this approach is the BufferedFrameDecryptor isn't even constructed if FrameEncryption is not in use. I have decided against merging the RtpReferenceFinder and EncryptedFrame stash because it introduced a lot of complexity around the mixed scenario where some of the frames in the stash are encrypted and others are not. In this case we would need to mark certain frames as decrypted which appeared to introduce more complexity than this simple decoupling. Bug: webrtc:10022 Change-Id: Iab74f7b7d25ef1cdd15c4a76b5daae1cfa24932c Reviewed-on: https://webrtc-review.googlesource.com/c/112221 Commit-Queue: Benjamin Wright <benwright@webrtc.org> Reviewed-by: Philip Eliasson <philipel@webrtc.org> Reviewed-by: Stefan Holmer <stefan@webrtc.org> Cr-Commit-Position: refs/heads/master@{#25865}
2018-11-30 16:18:26 -08:00
// Determines whether the frame should be stashed, dropped or handed off to
// the OnDecryptedFrameCallback.
void ManageEncryptedFrame(std::unique_ptr<RtpFrameObject> encrypted_frame);
Add BufferedFrameDecryptor to cleanly deal with receiving encrypted frames. This change introduces a new class BufferedFrameDecryptor that is responsible for decrypting received encrypted frames and passing them on to the RtpReferenceFinder. This decoupling refactoring was triggered by a new optimization also introduced in this patch to stash a small number of undecryptable frames if no frames have ever been decrypted. The goal of this optimization is to prevent re-fectching of key frames on low bandwidth networks simply because the key to decrypt them had not arrived yet. The optimization will stash 24 frames (about 1 second of video) in a ring buffer and will attempt to re-decrypt previously received frames on the first valid decryption. This allows the decoder to receive the key frame without having to request due to short key delivery latencies. In testing this is actually hit quite often and saves an entire RTT which can be up to 200ms on a bad network. As the scope of frame encryption increases in WebRTC and has more specialized optimizations that do not apply to the general flow it makes sense to move it to a more explicit bump in the stack protocol that is decoupled from the WebRTC main flow, similar to how SRTP is utilized with srtp_protect and srtp_unprotect. One advantage of this approach is the BufferedFrameDecryptor isn't even constructed if FrameEncryption is not in use. I have decided against merging the RtpReferenceFinder and EncryptedFrame stash because it introduced a lot of complexity around the mixed scenario where some of the frames in the stash are encrypted and others are not. In this case we would need to mark certain frames as decrypted which appeared to introduce more complexity than this simple decoupling. Bug: webrtc:10022 Change-Id: Iab74f7b7d25ef1cdd15c4a76b5daae1cfa24932c Reviewed-on: https://webrtc-review.googlesource.com/c/112221 Commit-Queue: Benjamin Wright <benwright@webrtc.org> Reviewed-by: Philip Eliasson <philipel@webrtc.org> Reviewed-by: Stefan Holmer <stefan@webrtc.org> Cr-Commit-Position: refs/heads/master@{#25865}
2018-11-30 16:18:26 -08:00
private:
// Represents what should be done with a given frame.
enum class FrameDecision { kStash, kDecrypted, kDrop };
// Attempts to decrypt the frame, if it fails and no prior frames have been
// decrypted it will return kStash. Otherwise fail to decrypts will return
// kDrop. Successful decryptions will always return kDecrypted.
FrameDecision DecryptFrame(RtpFrameObject* frame);
Add BufferedFrameDecryptor to cleanly deal with receiving encrypted frames. This change introduces a new class BufferedFrameDecryptor that is responsible for decrypting received encrypted frames and passing them on to the RtpReferenceFinder. This decoupling refactoring was triggered by a new optimization also introduced in this patch to stash a small number of undecryptable frames if no frames have ever been decrypted. The goal of this optimization is to prevent re-fectching of key frames on low bandwidth networks simply because the key to decrypt them had not arrived yet. The optimization will stash 24 frames (about 1 second of video) in a ring buffer and will attempt to re-decrypt previously received frames on the first valid decryption. This allows the decoder to receive the key frame without having to request due to short key delivery latencies. In testing this is actually hit quite often and saves an entire RTT which can be up to 200ms on a bad network. As the scope of frame encryption increases in WebRTC and has more specialized optimizations that do not apply to the general flow it makes sense to move it to a more explicit bump in the stack protocol that is decoupled from the WebRTC main flow, similar to how SRTP is utilized with srtp_protect and srtp_unprotect. One advantage of this approach is the BufferedFrameDecryptor isn't even constructed if FrameEncryption is not in use. I have decided against merging the RtpReferenceFinder and EncryptedFrame stash because it introduced a lot of complexity around the mixed scenario where some of the frames in the stash are encrypted and others are not. In this case we would need to mark certain frames as decrypted which appeared to introduce more complexity than this simple decoupling. Bug: webrtc:10022 Change-Id: Iab74f7b7d25ef1cdd15c4a76b5daae1cfa24932c Reviewed-on: https://webrtc-review.googlesource.com/c/112221 Commit-Queue: Benjamin Wright <benwright@webrtc.org> Reviewed-by: Philip Eliasson <philipel@webrtc.org> Reviewed-by: Stefan Holmer <stefan@webrtc.org> Cr-Commit-Position: refs/heads/master@{#25865}
2018-11-30 16:18:26 -08:00
// Retries all the stashed frames this is triggered each time a kDecrypted
// event occurs.
void RetryStashedFrames();
static const size_t kMaxStashedFrames = 24;
const bool generic_descriptor_auth_experiment_;
Add BufferedFrameDecryptor to cleanly deal with receiving encrypted frames. This change introduces a new class BufferedFrameDecryptor that is responsible for decrypting received encrypted frames and passing them on to the RtpReferenceFinder. This decoupling refactoring was triggered by a new optimization also introduced in this patch to stash a small number of undecryptable frames if no frames have ever been decrypted. The goal of this optimization is to prevent re-fectching of key frames on low bandwidth networks simply because the key to decrypt them had not arrived yet. The optimization will stash 24 frames (about 1 second of video) in a ring buffer and will attempt to re-decrypt previously received frames on the first valid decryption. This allows the decoder to receive the key frame without having to request due to short key delivery latencies. In testing this is actually hit quite often and saves an entire RTT which can be up to 200ms on a bad network. As the scope of frame encryption increases in WebRTC and has more specialized optimizations that do not apply to the general flow it makes sense to move it to a more explicit bump in the stack protocol that is decoupled from the WebRTC main flow, similar to how SRTP is utilized with srtp_protect and srtp_unprotect. One advantage of this approach is the BufferedFrameDecryptor isn't even constructed if FrameEncryption is not in use. I have decided against merging the RtpReferenceFinder and EncryptedFrame stash because it introduced a lot of complexity around the mixed scenario where some of the frames in the stash are encrypted and others are not. In this case we would need to mark certain frames as decrypted which appeared to introduce more complexity than this simple decoupling. Bug: webrtc:10022 Change-Id: Iab74f7b7d25ef1cdd15c4a76b5daae1cfa24932c Reviewed-on: https://webrtc-review.googlesource.com/c/112221 Commit-Queue: Benjamin Wright <benwright@webrtc.org> Reviewed-by: Philip Eliasson <philipel@webrtc.org> Reviewed-by: Stefan Holmer <stefan@webrtc.org> Cr-Commit-Position: refs/heads/master@{#25865}
2018-11-30 16:18:26 -08:00
bool first_frame_decrypted_ = false;
Reland "Refactor FrameDecryptorInterface::Decrypt to use new API." This reverts commit 7dd83e2bf73a7f1746c5ee976939bf52e19fa8be. Reason for revert: This wasn't the cause of the break. Original change's description: > Revert "Refactor FrameDecryptorInterface::Decrypt to use new API." > > This reverts commit 642aa81f7d5cc55d5b99e2abc51327eed9d40195. > > Reason for revert: Speculative revert. The chromium roll is failing: > https://ci.chromium.org/p/chromium/builders/try/linux-rel/64388 > But I can't figure out exactly what is failing, this looks suspecious. > > Original change's description: > > Refactor FrameDecryptorInterface::Decrypt to use new API. > > > > This change refactors the FrameDecryptorInterface to use the new API. The new > > API surface simply moves bytes_written to the return type and implements a > > simple Status type. > > > > Bug: webrtc:10512 > > Change-Id: I622c5d344d58e618853c94c2f691cf7c8fb73a36 > > Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/131460 > > Reviewed-by: Steve Anton <steveanton@webrtc.org> > > Reviewed-by: Fredrik Solenberg <solenberg@webrtc.org> > > Reviewed-by: Rasmus Brandt <brandtr@webrtc.org> > > Reviewed-by: Stefan Holmer <stefan@webrtc.org> > > Commit-Queue: Benjamin Wright <benwright@webrtc.org> > > Cr-Commit-Position: refs/heads/master@{#27497} > > TBR=brandtr@webrtc.org,steveanton@webrtc.org,solenberg@webrtc.org,ossu@webrtc.org,stefan@webrtc.org,benwright@webrtc.org > > Change-Id: Ia9ec70263762c34671af13f0d519e636eb8473cd > No-Presubmit: true > No-Tree-Checks: true > No-Try: true > Bug: webrtc:10512 > Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/132013 > Reviewed-by: Henrik Boström <hbos@webrtc.org> > Commit-Queue: Henrik Boström <hbos@webrtc.org> > Cr-Commit-Position: refs/heads/master@{#27510} TBR=brandtr@webrtc.org,steveanton@webrtc.org,solenberg@webrtc.org,hbos@webrtc.org,ossu@webrtc.org,stefan@webrtc.org,benwright@webrtc.org Change-Id: I8e4b7965cf1d1a1554c3b46e6245f5ad0d2dcbb4 No-Presubmit: true No-Tree-Checks: true No-Try: true Bug: webrtc:10512 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/131982 Reviewed-by: Benjamin Wright <benwright@webrtc.org> Commit-Queue: Benjamin Wright <benwright@webrtc.org> Cr-Commit-Position: refs/heads/master@{#27529}
2019-04-09 20:08:41 +00:00
FrameDecryptorInterface::Status last_status_ =
FrameDecryptorInterface::Status::kUnknown;
rtc::scoped_refptr<FrameDecryptorInterface> frame_decryptor_;
Add BufferedFrameDecryptor to cleanly deal with receiving encrypted frames. This change introduces a new class BufferedFrameDecryptor that is responsible for decrypting received encrypted frames and passing them on to the RtpReferenceFinder. This decoupling refactoring was triggered by a new optimization also introduced in this patch to stash a small number of undecryptable frames if no frames have ever been decrypted. The goal of this optimization is to prevent re-fectching of key frames on low bandwidth networks simply because the key to decrypt them had not arrived yet. The optimization will stash 24 frames (about 1 second of video) in a ring buffer and will attempt to re-decrypt previously received frames on the first valid decryption. This allows the decoder to receive the key frame without having to request due to short key delivery latencies. In testing this is actually hit quite often and saves an entire RTT which can be up to 200ms on a bad network. As the scope of frame encryption increases in WebRTC and has more specialized optimizations that do not apply to the general flow it makes sense to move it to a more explicit bump in the stack protocol that is decoupled from the WebRTC main flow, similar to how SRTP is utilized with srtp_protect and srtp_unprotect. One advantage of this approach is the BufferedFrameDecryptor isn't even constructed if FrameEncryption is not in use. I have decided against merging the RtpReferenceFinder and EncryptedFrame stash because it introduced a lot of complexity around the mixed scenario where some of the frames in the stash are encrypted and others are not. In this case we would need to mark certain frames as decrypted which appeared to introduce more complexity than this simple decoupling. Bug: webrtc:10022 Change-Id: Iab74f7b7d25ef1cdd15c4a76b5daae1cfa24932c Reviewed-on: https://webrtc-review.googlesource.com/c/112221 Commit-Queue: Benjamin Wright <benwright@webrtc.org> Reviewed-by: Philip Eliasson <philipel@webrtc.org> Reviewed-by: Stefan Holmer <stefan@webrtc.org> Cr-Commit-Position: refs/heads/master@{#25865}
2018-11-30 16:18:26 -08:00
OnDecryptedFrameCallback* const decrypted_frame_callback_;
OnDecryptionStatusChangeCallback* const decryption_status_change_callback_;
std::deque<std::unique_ptr<RtpFrameObject>> stashed_frames_;
Add BufferedFrameDecryptor to cleanly deal with receiving encrypted frames. This change introduces a new class BufferedFrameDecryptor that is responsible for decrypting received encrypted frames and passing them on to the RtpReferenceFinder. This decoupling refactoring was triggered by a new optimization also introduced in this patch to stash a small number of undecryptable frames if no frames have ever been decrypted. The goal of this optimization is to prevent re-fectching of key frames on low bandwidth networks simply because the key to decrypt them had not arrived yet. The optimization will stash 24 frames (about 1 second of video) in a ring buffer and will attempt to re-decrypt previously received frames on the first valid decryption. This allows the decoder to receive the key frame without having to request due to short key delivery latencies. In testing this is actually hit quite often and saves an entire RTT which can be up to 200ms on a bad network. As the scope of frame encryption increases in WebRTC and has more specialized optimizations that do not apply to the general flow it makes sense to move it to a more explicit bump in the stack protocol that is decoupled from the WebRTC main flow, similar to how SRTP is utilized with srtp_protect and srtp_unprotect. One advantage of this approach is the BufferedFrameDecryptor isn't even constructed if FrameEncryption is not in use. I have decided against merging the RtpReferenceFinder and EncryptedFrame stash because it introduced a lot of complexity around the mixed scenario where some of the frames in the stash are encrypted and others are not. In this case we would need to mark certain frames as decrypted which appeared to introduce more complexity than this simple decoupling. Bug: webrtc:10022 Change-Id: Iab74f7b7d25ef1cdd15c4a76b5daae1cfa24932c Reviewed-on: https://webrtc-review.googlesource.com/c/112221 Commit-Queue: Benjamin Wright <benwright@webrtc.org> Reviewed-by: Philip Eliasson <philipel@webrtc.org> Reviewed-by: Stefan Holmer <stefan@webrtc.org> Cr-Commit-Position: refs/heads/master@{#25865}
2018-11-30 16:18:26 -08:00
};
} // namespace webrtc
#endif // VIDEO_BUFFERED_FRAME_DECRYPTOR_H_