admin/webrtc_m130
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
webrtc_m130/call/adaptation/resource.cc

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

103 lines
3.1 KiB
C++
Raw Normal View History

Initial version of ResourceAdaptationProcessor and friends. This CL adds Resource, ResourceConsumer, ResourceConsumerConfiguration and ResourceAdaptationProcessor and implements the algorithm outlined in https://docs.google.com/presentation/d/13jyqCWNpIa873iKT6yDuB5Q5ma-c0CvxBpX--0tCclY/edit?usp=sharing. Simply put, if any resource (such as "CPU") is overusing, the most expensive consumer (e.g. encoded stream) is adapted one step down. If all resources are underusing, the least expensive consumer is adapted one step up. The current resources, consumers and configurations are all fakes; this CL has no effect on the current adaptation algorithms used in practise, but it lays down the foundation for future work in this area. Bug: webrtc:11167, webrtc:11168, webrtc:11169 Change-Id: I4054ec7728a52a49e137eee6fa67fa27debd9254 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/161237 Commit-Queue: Henrik Boström <hbos@webrtc.org> Reviewed-by: Evan Shrubsole <eshr@google.com> Reviewed-by: Stefan Holmer <stefan@webrtc.org> Reviewed-by: Erik Språng <sprang@webrtc.org> Cr-Commit-Position: refs/heads/master@{#30053}
2019-12-10 14:14:09 +01:00
/*
* Copyright 2019 The WebRTC Project Authors. All rights reserved.
*
* Use of this source code is governed by a BSD-style license
* that can be found in the LICENSE file in the root of the source
* tree. An additional intellectual property rights grant can be found
* in the file PATENTS. All contributing project authors may
* be found in the AUTHORS file in the root of the source tree.
*/
#include "call/adaptation/resource.h"
Support injecting new Resources for overuse * This replaces the video stream methods for forcing adaptation with a mock resource that triggers overuse. * Resources can now be injected to the Module using the AddResource function. * Resources now have tests for adding and removing callbacks. * Quality/EncoderUse% resources are tracked in the Resource list of the adaptation module. * The adaptation module ties all resources to a reason to keep stats working as expected. BUG=webrtc:11377 Change-Id: I1f5902f7416dc41b4915c0072e6f0da2bb3bb2b7 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/168948 Commit-Queue: Evan Shrubsole <eshr@google.com> Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org> Reviewed-by: Henrik Boström <hbos@webrtc.org> Cr-Commit-Position: refs/heads/master@{#30610}
2020-02-25 16:26:01 +01:00
#include "absl/algorithm/container.h"
[Overuse] Implement Resource and ResourceUsageListener. The Resource interface (previously a skeleton not used outside of testing) is updated to inform listeners of changes to resource usage. Debugging methods are removed (Name, UsageUnitsOfMeasurements, CurrentUsage). The interface is implemented by OveruseFrameDetectorResourceAdaptationModule's inner classes EncodeUsageResource and QualityScalerResource. The new ResourceUsageListener interface is implemented by OveruseFrameDetectorResourceAdaptationModule. In order to avoid adding AdaptationObserverInterface::AdaptReason to the ResourceUsageListener interface, the module figures out if the reason is "kCpu" or "kQuality" by looking which Resource object triggered OnResourceUsageStateMeasured(). These resources no longer need an explicit reference to OveruseFrameDetectorResourceAdaptationModule and could potentially be used by a different module. In this CL, AdaptationObserverInterface::AdaptDown()'s return value is still needed by QualityScaler. This is mirrored in the return value of ResourceUsageListener::OnResourceUsageStateMeasured(). A TODO is added to remove it and a comment explains how the current implementation seems to break the contract of the method (as was the case prior to this CL). Follow-up work include: - Move EncodeUsageResource and QualityScalerResource to separate files. - Make resources injectable, allowing fake resources in testing and removing OnResourceOveruseForTesting() methods. (Investigate adding the necessary input signals to the Resource interface or relevant sub-interfaces so that the module does not need to know which Resource implementation is used.) - And more! See whiteboard :) Bug: webrtc:11222 Change-Id: I0a46ace4a2e617874e3ee97e67e3a199fef420a2 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/168180 Commit-Queue: Henrik Boström <hbos@webrtc.org> Reviewed-by: Erik Språng <sprang@webrtc.org> Reviewed-by: Evan Shrubsole <eshr@google.com> Cr-Commit-Position: refs/heads/master@{#30469}
2020-02-06 12:49:57 +01:00
#include "rtc_base/checks.h"
Initial version of ResourceAdaptationProcessor and friends. This CL adds Resource, ResourceConsumer, ResourceConsumerConfiguration and ResourceAdaptationProcessor and implements the algorithm outlined in https://docs.google.com/presentation/d/13jyqCWNpIa873iKT6yDuB5Q5ma-c0CvxBpX--0tCclY/edit?usp=sharing. Simply put, if any resource (such as "CPU") is overusing, the most expensive consumer (e.g. encoded stream) is adapted one step down. If all resources are underusing, the least expensive consumer is adapted one step up. The current resources, consumers and configurations are all fakes; this CL has no effect on the current adaptation algorithms used in practise, but it lays down the foundation for future work in this area. Bug: webrtc:11167, webrtc:11168, webrtc:11169 Change-Id: I4054ec7728a52a49e137eee6fa67fa27debd9254 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/161237 Commit-Queue: Henrik Boström <hbos@webrtc.org> Reviewed-by: Evan Shrubsole <eshr@google.com> Reviewed-by: Stefan Holmer <stefan@webrtc.org> Reviewed-by: Erik Språng <sprang@webrtc.org> Cr-Commit-Position: refs/heads/master@{#30053}
2019-12-10 14:14:09 +01:00
namespace webrtc {
[Adaptation] Add more ResourceAdaptationProcessor logging. This should help debugging when adaptation is or is not happening unexpectedly. Log spam is prevented by not logging if the same result happened to the same resource already and we haven't adapted since then. Bug: webrtc:11616 Change-Id: Ia6c5cc35061d252f1c66f2f2bf3b94d2485498d6 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/176221 Commit-Queue: Henrik Boström <hbos@webrtc.org> Reviewed-by: Evan Shrubsole <eshr@google.com> Reviewed-by: Harald Alvestrand <hta@webrtc.org> Cr-Commit-Position: refs/heads/master@{#31378}
2020-05-28 16:22:42 +02:00
const char* ResourceUsageStateToString(ResourceUsageState usage_state) {
switch (usage_state) {
case ResourceUsageState::kOveruse:
return "kOveruse";
case ResourceUsageState::kUnderuse:
return "kUnderuse";
}
}
[Overuse] Implement Resource and ResourceUsageListener. The Resource interface (previously a skeleton not used outside of testing) is updated to inform listeners of changes to resource usage. Debugging methods are removed (Name, UsageUnitsOfMeasurements, CurrentUsage). The interface is implemented by OveruseFrameDetectorResourceAdaptationModule's inner classes EncodeUsageResource and QualityScalerResource. The new ResourceUsageListener interface is implemented by OveruseFrameDetectorResourceAdaptationModule. In order to avoid adding AdaptationObserverInterface::AdaptReason to the ResourceUsageListener interface, the module figures out if the reason is "kCpu" or "kQuality" by looking which Resource object triggered OnResourceUsageStateMeasured(). These resources no longer need an explicit reference to OveruseFrameDetectorResourceAdaptationModule and could potentially be used by a different module. In this CL, AdaptationObserverInterface::AdaptDown()'s return value is still needed by QualityScaler. This is mirrored in the return value of ResourceUsageListener::OnResourceUsageStateMeasured(). A TODO is added to remove it and a comment explains how the current implementation seems to break the contract of the method (as was the case prior to this CL). Follow-up work include: - Move EncodeUsageResource and QualityScalerResource to separate files. - Make resources injectable, allowing fake resources in testing and removing OnResourceOveruseForTesting() methods. (Investigate adding the necessary input signals to the Resource interface or relevant sub-interfaces so that the module does not need to know which Resource implementation is used.) - And more! See whiteboard :) Bug: webrtc:11222 Change-Id: I0a46ace4a2e617874e3ee97e67e3a199fef420a2 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/168180 Commit-Queue: Henrik Boström <hbos@webrtc.org> Reviewed-by: Erik Språng <sprang@webrtc.org> Reviewed-by: Evan Shrubsole <eshr@google.com> Cr-Commit-Position: refs/heads/master@{#30469}
2020-02-06 12:49:57 +01:00
ResourceListener::~ResourceListener() {}
Initial version of ResourceAdaptationProcessor and friends. This CL adds Resource, ResourceConsumer, ResourceConsumerConfiguration and ResourceAdaptationProcessor and implements the algorithm outlined in https://docs.google.com/presentation/d/13jyqCWNpIa873iKT6yDuB5Q5ma-c0CvxBpX--0tCclY/edit?usp=sharing. Simply put, if any resource (such as "CPU") is overusing, the most expensive consumer (e.g. encoded stream) is adapted one step down. If all resources are underusing, the least expensive consumer is adapted one step up. The current resources, consumers and configurations are all fakes; this CL has no effect on the current adaptation algorithms used in practise, but it lays down the foundation for future work in this area. Bug: webrtc:11167, webrtc:11168, webrtc:11169 Change-Id: I4054ec7728a52a49e137eee6fa67fa27debd9254 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/161237 Commit-Queue: Henrik Boström <hbos@webrtc.org> Reviewed-by: Evan Shrubsole <eshr@google.com> Reviewed-by: Stefan Holmer <stefan@webrtc.org> Reviewed-by: Erik Språng <sprang@webrtc.org> Cr-Commit-Position: refs/heads/master@{#30053}
2019-12-10 14:14:09 +01:00
[Adaptation] Move adaptation logic to a separate task queue. This CL unblocks future Call-Level Mitigation strategies by moving the ResourceAdaptationProcessor to a separate task queue. This signifies a major milestone in the new resource adaptation architecture because with this CL the threading model is in place and moving the Processor to the Call and increasing its responsibilities is made possible. In this CL, we still have one Processor per VideoStreamEncoder and the VideoStreamEncoder is responsible for the creation and the destruction of its Processor and that Processor's task queue. But the PostTasks are in place and the decision-making is executed on a separate queue. This CL: - Moves ResourceAdaptationProcessor to an adaptation task queue. It continues to be entirely single-threaded, but now operates on a separate task queue. - Makes Resources thread-safe: Interaction with the Processor, i.e. OnResourceUsageStateMeasured() and IsAdaptationUpAllowed(), happens on the adaptation task queue. State updates are pushed from the encoder task queue with PostTasks. - QualityScalerResource operates on both task queues; the QP usage callbacks are invoked asynchronously. - The VideoStreamEncoderResourceManager operates on the encoder task queue with the following exceptions: 1) Its resources are accessible on any thread (using a mutex). This is OK because resources are reference counted and thread safe. This aids adding and removing resources to the Processor on the adaptation task queue. 2) |active_counts_| is moved to the adaptation task queue. This makes it possible for PreventAdaptUpDueToActiveCounts to run IsAdaptationUpAllowed() on the adaptation task queue. A side-effect of this is that some stats reporting now happen on the adaptation task queue, but that is OK because VideoStreamEncoderObserver is thread-safe. The Manager is updated to take the new threading model into account: - OnFrameDroppedDueToSize() posts to the adaptation task queue to invoke the Processor. - OnVideoSourceRestrictionsUpdated(), now invoked on the adaptation task queue, updates |active_counts_| synchronously but posts to the encoder task queue to update video source restrictions (which it only uses to calculate target frame rate). - MaybePerformQualityRampupExperiment() posts to the adaptation task queue to maybe reset video source restrictions on the Processor. |quality_rampup_done_| is made std::atomic. Bug: webrtc:11542, webrtc:11520 Change-Id: I1cfd76e0cd42f006a6d2527f5aa2aeb5266ba6d6 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/174441 Reviewed-by: Evan Shrubsole <eshr@google.com> Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org> Commit-Queue: Henrik Boström <hbos@webrtc.org> Cr-Commit-Position: refs/heads/master@{#31231}
2020-05-12 18:49:07 +02:00
Resource::Resource()
: encoder_queue_(nullptr),
resource_adaptation_queue_(nullptr),
usage_state_(absl::nullopt),
listener_(nullptr) {}
Initial version of ResourceAdaptationProcessor and friends. This CL adds Resource, ResourceConsumer, ResourceConsumerConfiguration and ResourceAdaptationProcessor and implements the algorithm outlined in https://docs.google.com/presentation/d/13jyqCWNpIa873iKT6yDuB5Q5ma-c0CvxBpX--0tCclY/edit?usp=sharing. Simply put, if any resource (such as "CPU") is overusing, the most expensive consumer (e.g. encoded stream) is adapted one step down. If all resources are underusing, the least expensive consumer is adapted one step up. The current resources, consumers and configurations are all fakes; this CL has no effect on the current adaptation algorithms used in practise, but it lays down the foundation for future work in this area. Bug: webrtc:11167, webrtc:11168, webrtc:11169 Change-Id: I4054ec7728a52a49e137eee6fa67fa27debd9254 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/161237 Commit-Queue: Henrik Boström <hbos@webrtc.org> Reviewed-by: Evan Shrubsole <eshr@google.com> Reviewed-by: Stefan Holmer <stefan@webrtc.org> Reviewed-by: Erik Språng <sprang@webrtc.org> Cr-Commit-Position: refs/heads/master@{#30053}
2019-12-10 14:14:09 +01:00
[Adaptation] Make Resources reference counted and add more DCHECKs. In a future CL, adaptation processing and stream encoder resource management will happen on different task queues. When this is the case, asynchronous tasks will be posted in both directions and some resources will have internal states used on multiple threads. This CL makes the Resource class reference counted in order to support posting tasks to a different threads without risk of use-after-free when a posted task is executed with a delay. This is preferred over WeakPtr strategies because WeakPtrs are single-threaded and preferred over raw pointer usage because the reference counted approach enables more compile-time and run-time assurance. This is also "future proof"; when resources can be injected through public APIs, ownership needs to be shared between libwebrtc and the application (e.g. Chrome). To reduce the risk of making mistakes in the future CL, sequence checkers and task queue DCHECKs are added as well as other DCHECKs to make sure things have been cleaned up before destruction, e.g: - Processor gets a sequence checker. It is entirely single-threaded. - Processor must not have any attached listeners or resources on destruction. - Resources must not have any listeners on destruction. - The Manager, EncodeUsageResource and QualityScalerResource DCHECKs they are running on the encoder queue. - TODOs are added illustrating where we want to add PostTasks in the future CL. Lastly, upon VideoStreamEncoder::Stop() we delete the ResourceAdaptationProcessor. Because the Processor is already used in posted tasks, some if statements are added to ensure the Processor is not used after destruction. Bug: webrtc:11542, webrtc:11520 Change-Id: Ibaa8a61d86d87a71f477d1075a117c28d9d2d285 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/174760 Commit-Queue: Henrik Boström <hbos@webrtc.org> Reviewed-by: Evan Shrubsole <eshr@google.com> Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org> Cr-Commit-Position: refs/heads/master@{#31217}
2020-05-11 16:29:22 +02:00
Resource::~Resource() {
RTC_DCHECK(!listener_)
<< "There is a listener depending on a Resource being destroyed.";
}
Initial version of ResourceAdaptationProcessor and friends. This CL adds Resource, ResourceConsumer, ResourceConsumerConfiguration and ResourceAdaptationProcessor and implements the algorithm outlined in https://docs.google.com/presentation/d/13jyqCWNpIa873iKT6yDuB5Q5ma-c0CvxBpX--0tCclY/edit?usp=sharing. Simply put, if any resource (such as "CPU") is overusing, the most expensive consumer (e.g. encoded stream) is adapted one step down. If all resources are underusing, the least expensive consumer is adapted one step up. The current resources, consumers and configurations are all fakes; this CL has no effect on the current adaptation algorithms used in practise, but it lays down the foundation for future work in this area. Bug: webrtc:11167, webrtc:11168, webrtc:11169 Change-Id: I4054ec7728a52a49e137eee6fa67fa27debd9254 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/161237 Commit-Queue: Henrik Boström <hbos@webrtc.org> Reviewed-by: Evan Shrubsole <eshr@google.com> Reviewed-by: Stefan Holmer <stefan@webrtc.org> Reviewed-by: Erik Språng <sprang@webrtc.org> Cr-Commit-Position: refs/heads/master@{#30053}
2019-12-10 14:14:09 +01:00
[Adaptation] Move adaptation logic to a separate task queue. This CL unblocks future Call-Level Mitigation strategies by moving the ResourceAdaptationProcessor to a separate task queue. This signifies a major milestone in the new resource adaptation architecture because with this CL the threading model is in place and moving the Processor to the Call and increasing its responsibilities is made possible. In this CL, we still have one Processor per VideoStreamEncoder and the VideoStreamEncoder is responsible for the creation and the destruction of its Processor and that Processor's task queue. But the PostTasks are in place and the decision-making is executed on a separate queue. This CL: - Moves ResourceAdaptationProcessor to an adaptation task queue. It continues to be entirely single-threaded, but now operates on a separate task queue. - Makes Resources thread-safe: Interaction with the Processor, i.e. OnResourceUsageStateMeasured() and IsAdaptationUpAllowed(), happens on the adaptation task queue. State updates are pushed from the encoder task queue with PostTasks. - QualityScalerResource operates on both task queues; the QP usage callbacks are invoked asynchronously. - The VideoStreamEncoderResourceManager operates on the encoder task queue with the following exceptions: 1) Its resources are accessible on any thread (using a mutex). This is OK because resources are reference counted and thread safe. This aids adding and removing resources to the Processor on the adaptation task queue. 2) |active_counts_| is moved to the adaptation task queue. This makes it possible for PreventAdaptUpDueToActiveCounts to run IsAdaptationUpAllowed() on the adaptation task queue. A side-effect of this is that some stats reporting now happen on the adaptation task queue, but that is OK because VideoStreamEncoderObserver is thread-safe. The Manager is updated to take the new threading model into account: - OnFrameDroppedDueToSize() posts to the adaptation task queue to invoke the Processor. - OnVideoSourceRestrictionsUpdated(), now invoked on the adaptation task queue, updates |active_counts_| synchronously but posts to the encoder task queue to update video source restrictions (which it only uses to calculate target frame rate). - MaybePerformQualityRampupExperiment() posts to the adaptation task queue to maybe reset video source restrictions on the Processor. |quality_rampup_done_| is made std::atomic. Bug: webrtc:11542, webrtc:11520 Change-Id: I1cfd76e0cd42f006a6d2527f5aa2aeb5266ba6d6 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/174441 Reviewed-by: Evan Shrubsole <eshr@google.com> Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org> Commit-Queue: Henrik Boström <hbos@webrtc.org> Cr-Commit-Position: refs/heads/master@{#31231}
2020-05-12 18:49:07 +02:00
void Resource::Initialize(rtc::TaskQueue* encoder_queue,
rtc::TaskQueue* resource_adaptation_queue) {
RTC_DCHECK(!encoder_queue_);
RTC_DCHECK(encoder_queue);
RTC_DCHECK(!resource_adaptation_queue_);
RTC_DCHECK(resource_adaptation_queue);
encoder_queue_ = encoder_queue;
resource_adaptation_queue_ = resource_adaptation_queue;
}
[Adaptation] Make ResourceUsageState nullable, remove kStable. This CL is part of the Call-Level Adaptation Processing design doc: https://docs.google.com/document/d/1ZyC26yOCknrrcYa839ZWLxD6o6Gig5A3lVTh4E41074/edit?usp=sharing The ResourceUsageState was written as: {kOveruse, kStable, kUnderuse}. The assumption was that if a resource neither wanted to adapt up or down it would report kStable. But with the addition of Resource::IsAdaptationUpAllowed() (prior CL) the notion of being "stable" was already captured outside of ResourceUsageState. Furthermore, kStable failed to capture what IsAdaptationUpAllowed() did not: whether we can go up depends on the resulting resolution or frame rate (restrictions_after). Perhaps we can go up a little, but not a lot. This CL also adds Resource::ClearUsageState(). After applying an adaptation, all usage states become invalidated (new measurements are needed to know if we are still over- or underusing). This was always the case, but prior to this CL this was not accurately reflected in the Resource::usage_state() in-between measurements. Bug: webrtc:11172 Change-Id: I140ff3114025b7732e530564690783e168d2509b Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/173088 Reviewed-by: Evan Shrubsole <eshr@google.com> Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org> Commit-Queue: Henrik Boström <hbos@webrtc.org> Cr-Commit-Position: refs/heads/master@{#31110}
2020-04-20 12:04:12 +02:00
void Resource::SetResourceListener(ResourceListener* listener) {
[Adaptation] Move adaptation logic to a separate task queue. This CL unblocks future Call-Level Mitigation strategies by moving the ResourceAdaptationProcessor to a separate task queue. This signifies a major milestone in the new resource adaptation architecture because with this CL the threading model is in place and moving the Processor to the Call and increasing its responsibilities is made possible. In this CL, we still have one Processor per VideoStreamEncoder and the VideoStreamEncoder is responsible for the creation and the destruction of its Processor and that Processor's task queue. But the PostTasks are in place and the decision-making is executed on a separate queue. This CL: - Moves ResourceAdaptationProcessor to an adaptation task queue. It continues to be entirely single-threaded, but now operates on a separate task queue. - Makes Resources thread-safe: Interaction with the Processor, i.e. OnResourceUsageStateMeasured() and IsAdaptationUpAllowed(), happens on the adaptation task queue. State updates are pushed from the encoder task queue with PostTasks. - QualityScalerResource operates on both task queues; the QP usage callbacks are invoked asynchronously. - The VideoStreamEncoderResourceManager operates on the encoder task queue with the following exceptions: 1) Its resources are accessible on any thread (using a mutex). This is OK because resources are reference counted and thread safe. This aids adding and removing resources to the Processor on the adaptation task queue. 2) |active_counts_| is moved to the adaptation task queue. This makes it possible for PreventAdaptUpDueToActiveCounts to run IsAdaptationUpAllowed() on the adaptation task queue. A side-effect of this is that some stats reporting now happen on the adaptation task queue, but that is OK because VideoStreamEncoderObserver is thread-safe. The Manager is updated to take the new threading model into account: - OnFrameDroppedDueToSize() posts to the adaptation task queue to invoke the Processor. - OnVideoSourceRestrictionsUpdated(), now invoked on the adaptation task queue, updates |active_counts_| synchronously but posts to the encoder task queue to update video source restrictions (which it only uses to calculate target frame rate). - MaybePerformQualityRampupExperiment() posts to the adaptation task queue to maybe reset video source restrictions on the Processor. |quality_rampup_done_| is made std::atomic. Bug: webrtc:11542, webrtc:11520 Change-Id: I1cfd76e0cd42f006a6d2527f5aa2aeb5266ba6d6 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/174441 Reviewed-by: Evan Shrubsole <eshr@google.com> Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org> Commit-Queue: Henrik Boström <hbos@webrtc.org> Cr-Commit-Position: refs/heads/master@{#31231}
2020-05-12 18:49:07 +02:00
RTC_DCHECK(resource_adaptation_queue_);
RTC_DCHECK_RUN_ON(resource_adaptation_queue_);
[Adaptation] Make ResourceUsageState nullable, remove kStable. This CL is part of the Call-Level Adaptation Processing design doc: https://docs.google.com/document/d/1ZyC26yOCknrrcYa839ZWLxD6o6Gig5A3lVTh4E41074/edit?usp=sharing The ResourceUsageState was written as: {kOveruse, kStable, kUnderuse}. The assumption was that if a resource neither wanted to adapt up or down it would report kStable. But with the addition of Resource::IsAdaptationUpAllowed() (prior CL) the notion of being "stable" was already captured outside of ResourceUsageState. Furthermore, kStable failed to capture what IsAdaptationUpAllowed() did not: whether we can go up depends on the resulting resolution or frame rate (restrictions_after). Perhaps we can go up a little, but not a lot. This CL also adds Resource::ClearUsageState(). After applying an adaptation, all usage states become invalidated (new measurements are needed to know if we are still over- or underusing). This was always the case, but prior to this CL this was not accurately reflected in the Resource::usage_state() in-between measurements. Bug: webrtc:11172 Change-Id: I140ff3114025b7732e530564690783e168d2509b Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/173088 Reviewed-by: Evan Shrubsole <eshr@google.com> Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org> Commit-Queue: Henrik Boström <hbos@webrtc.org> Cr-Commit-Position: refs/heads/master@{#31110}
2020-04-20 12:04:12 +02:00
// If you want to change listener you need to unregister the old listener by
// setting it to null first.
RTC_DCHECK(!listener_ || !listener) << "A listener is already set";
listener_ = listener;
[Overuse] Implement Resource and ResourceUsageListener. The Resource interface (previously a skeleton not used outside of testing) is updated to inform listeners of changes to resource usage. Debugging methods are removed (Name, UsageUnitsOfMeasurements, CurrentUsage). The interface is implemented by OveruseFrameDetectorResourceAdaptationModule's inner classes EncodeUsageResource and QualityScalerResource. The new ResourceUsageListener interface is implemented by OveruseFrameDetectorResourceAdaptationModule. In order to avoid adding AdaptationObserverInterface::AdaptReason to the ResourceUsageListener interface, the module figures out if the reason is "kCpu" or "kQuality" by looking which Resource object triggered OnResourceUsageStateMeasured(). These resources no longer need an explicit reference to OveruseFrameDetectorResourceAdaptationModule and could potentially be used by a different module. In this CL, AdaptationObserverInterface::AdaptDown()'s return value is still needed by QualityScaler. This is mirrored in the return value of ResourceUsageListener::OnResourceUsageStateMeasured(). A TODO is added to remove it and a comment explains how the current implementation seems to break the contract of the method (as was the case prior to this CL). Follow-up work include: - Move EncodeUsageResource and QualityScalerResource to separate files. - Make resources injectable, allowing fake resources in testing and removing OnResourceOveruseForTesting() methods. (Investigate adding the necessary input signals to the Resource interface or relevant sub-interfaces so that the module does not need to know which Resource implementation is used.) - And more! See whiteboard :) Bug: webrtc:11222 Change-Id: I0a46ace4a2e617874e3ee97e67e3a199fef420a2 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/168180 Commit-Queue: Henrik Boström <hbos@webrtc.org> Reviewed-by: Erik Språng <sprang@webrtc.org> Reviewed-by: Evan Shrubsole <eshr@google.com> Cr-Commit-Position: refs/heads/master@{#30469}
2020-02-06 12:49:57 +01:00
}
[Adaptation] Make ResourceUsageState nullable, remove kStable. This CL is part of the Call-Level Adaptation Processing design doc: https://docs.google.com/document/d/1ZyC26yOCknrrcYa839ZWLxD6o6Gig5A3lVTh4E41074/edit?usp=sharing The ResourceUsageState was written as: {kOveruse, kStable, kUnderuse}. The assumption was that if a resource neither wanted to adapt up or down it would report kStable. But with the addition of Resource::IsAdaptationUpAllowed() (prior CL) the notion of being "stable" was already captured outside of ResourceUsageState. Furthermore, kStable failed to capture what IsAdaptationUpAllowed() did not: whether we can go up depends on the resulting resolution or frame rate (restrictions_after). Perhaps we can go up a little, but not a lot. This CL also adds Resource::ClearUsageState(). After applying an adaptation, all usage states become invalidated (new measurements are needed to know if we are still over- or underusing). This was always the case, but prior to this CL this was not accurately reflected in the Resource::usage_state() in-between measurements. Bug: webrtc:11172 Change-Id: I140ff3114025b7732e530564690783e168d2509b Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/173088 Reviewed-by: Evan Shrubsole <eshr@google.com> Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org> Commit-Queue: Henrik Boström <hbos@webrtc.org> Cr-Commit-Position: refs/heads/master@{#31110}
2020-04-20 12:04:12 +02:00
absl::optional<ResourceUsageState> Resource::usage_state() const {
[Adaptation] Move adaptation logic to a separate task queue. This CL unblocks future Call-Level Mitigation strategies by moving the ResourceAdaptationProcessor to a separate task queue. This signifies a major milestone in the new resource adaptation architecture because with this CL the threading model is in place and moving the Processor to the Call and increasing its responsibilities is made possible. In this CL, we still have one Processor per VideoStreamEncoder and the VideoStreamEncoder is responsible for the creation and the destruction of its Processor and that Processor's task queue. But the PostTasks are in place and the decision-making is executed on a separate queue. This CL: - Moves ResourceAdaptationProcessor to an adaptation task queue. It continues to be entirely single-threaded, but now operates on a separate task queue. - Makes Resources thread-safe: Interaction with the Processor, i.e. OnResourceUsageStateMeasured() and IsAdaptationUpAllowed(), happens on the adaptation task queue. State updates are pushed from the encoder task queue with PostTasks. - QualityScalerResource operates on both task queues; the QP usage callbacks are invoked asynchronously. - The VideoStreamEncoderResourceManager operates on the encoder task queue with the following exceptions: 1) Its resources are accessible on any thread (using a mutex). This is OK because resources are reference counted and thread safe. This aids adding and removing resources to the Processor on the adaptation task queue. 2) |active_counts_| is moved to the adaptation task queue. This makes it possible for PreventAdaptUpDueToActiveCounts to run IsAdaptationUpAllowed() on the adaptation task queue. A side-effect of this is that some stats reporting now happen on the adaptation task queue, but that is OK because VideoStreamEncoderObserver is thread-safe. The Manager is updated to take the new threading model into account: - OnFrameDroppedDueToSize() posts to the adaptation task queue to invoke the Processor. - OnVideoSourceRestrictionsUpdated(), now invoked on the adaptation task queue, updates |active_counts_| synchronously but posts to the encoder task queue to update video source restrictions (which it only uses to calculate target frame rate). - MaybePerformQualityRampupExperiment() posts to the adaptation task queue to maybe reset video source restrictions on the Processor. |quality_rampup_done_| is made std::atomic. Bug: webrtc:11542, webrtc:11520 Change-Id: I1cfd76e0cd42f006a6d2527f5aa2aeb5266ba6d6 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/174441 Reviewed-by: Evan Shrubsole <eshr@google.com> Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org> Commit-Queue: Henrik Boström <hbos@webrtc.org> Cr-Commit-Position: refs/heads/master@{#31231}
2020-05-12 18:49:07 +02:00
RTC_DCHECK(resource_adaptation_queue_);
RTC_DCHECK_RUN_ON(resource_adaptation_queue_);
[Adaptation] Make ResourceUsageState nullable, remove kStable. This CL is part of the Call-Level Adaptation Processing design doc: https://docs.google.com/document/d/1ZyC26yOCknrrcYa839ZWLxD6o6Gig5A3lVTh4E41074/edit?usp=sharing The ResourceUsageState was written as: {kOveruse, kStable, kUnderuse}. The assumption was that if a resource neither wanted to adapt up or down it would report kStable. But with the addition of Resource::IsAdaptationUpAllowed() (prior CL) the notion of being "stable" was already captured outside of ResourceUsageState. Furthermore, kStable failed to capture what IsAdaptationUpAllowed() did not: whether we can go up depends on the resulting resolution or frame rate (restrictions_after). Perhaps we can go up a little, but not a lot. This CL also adds Resource::ClearUsageState(). After applying an adaptation, all usage states become invalidated (new measurements are needed to know if we are still over- or underusing). This was always the case, but prior to this CL this was not accurately reflected in the Resource::usage_state() in-between measurements. Bug: webrtc:11172 Change-Id: I140ff3114025b7732e530564690783e168d2509b Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/173088 Reviewed-by: Evan Shrubsole <eshr@google.com> Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org> Commit-Queue: Henrik Boström <hbos@webrtc.org> Cr-Commit-Position: refs/heads/master@{#31110}
2020-04-20 12:04:12 +02:00
return usage_state_;
Support injecting new Resources for overuse * This replaces the video stream methods for forcing adaptation with a mock resource that triggers overuse. * Resources can now be injected to the Module using the AddResource function. * Resources now have tests for adding and removing callbacks. * Quality/EncoderUse% resources are tracked in the Resource list of the adaptation module. * The adaptation module ties all resources to a reason to keep stats working as expected. BUG=webrtc:11377 Change-Id: I1f5902f7416dc41b4915c0072e6f0da2bb3bb2b7 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/168948 Commit-Queue: Evan Shrubsole <eshr@google.com> Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org> Reviewed-by: Henrik Boström <hbos@webrtc.org> Cr-Commit-Position: refs/heads/master@{#30610}
2020-02-25 16:26:01 +01:00
}
[Adaptation] Make ResourceUsageState nullable, remove kStable. This CL is part of the Call-Level Adaptation Processing design doc: https://docs.google.com/document/d/1ZyC26yOCknrrcYa839ZWLxD6o6Gig5A3lVTh4E41074/edit?usp=sharing The ResourceUsageState was written as: {kOveruse, kStable, kUnderuse}. The assumption was that if a resource neither wanted to adapt up or down it would report kStable. But with the addition of Resource::IsAdaptationUpAllowed() (prior CL) the notion of being "stable" was already captured outside of ResourceUsageState. Furthermore, kStable failed to capture what IsAdaptationUpAllowed() did not: whether we can go up depends on the resulting resolution or frame rate (restrictions_after). Perhaps we can go up a little, but not a lot. This CL also adds Resource::ClearUsageState(). After applying an adaptation, all usage states become invalidated (new measurements are needed to know if we are still over- or underusing). This was always the case, but prior to this CL this was not accurately reflected in the Resource::usage_state() in-between measurements. Bug: webrtc:11172 Change-Id: I140ff3114025b7732e530564690783e168d2509b Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/173088 Reviewed-by: Evan Shrubsole <eshr@google.com> Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org> Commit-Queue: Henrik Boström <hbos@webrtc.org> Cr-Commit-Position: refs/heads/master@{#31110}
2020-04-20 12:04:12 +02:00
void Resource::ClearUsageState() {
[Adaptation] Move adaptation logic to a separate task queue. This CL unblocks future Call-Level Mitigation strategies by moving the ResourceAdaptationProcessor to a separate task queue. This signifies a major milestone in the new resource adaptation architecture because with this CL the threading model is in place and moving the Processor to the Call and increasing its responsibilities is made possible. In this CL, we still have one Processor per VideoStreamEncoder and the VideoStreamEncoder is responsible for the creation and the destruction of its Processor and that Processor's task queue. But the PostTasks are in place and the decision-making is executed on a separate queue. This CL: - Moves ResourceAdaptationProcessor to an adaptation task queue. It continues to be entirely single-threaded, but now operates on a separate task queue. - Makes Resources thread-safe: Interaction with the Processor, i.e. OnResourceUsageStateMeasured() and IsAdaptationUpAllowed(), happens on the adaptation task queue. State updates are pushed from the encoder task queue with PostTasks. - QualityScalerResource operates on both task queues; the QP usage callbacks are invoked asynchronously. - The VideoStreamEncoderResourceManager operates on the encoder task queue with the following exceptions: 1) Its resources are accessible on any thread (using a mutex). This is OK because resources are reference counted and thread safe. This aids adding and removing resources to the Processor on the adaptation task queue. 2) |active_counts_| is moved to the adaptation task queue. This makes it possible for PreventAdaptUpDueToActiveCounts to run IsAdaptationUpAllowed() on the adaptation task queue. A side-effect of this is that some stats reporting now happen on the adaptation task queue, but that is OK because VideoStreamEncoderObserver is thread-safe. The Manager is updated to take the new threading model into account: - OnFrameDroppedDueToSize() posts to the adaptation task queue to invoke the Processor. - OnVideoSourceRestrictionsUpdated(), now invoked on the adaptation task queue, updates |active_counts_| synchronously but posts to the encoder task queue to update video source restrictions (which it only uses to calculate target frame rate). - MaybePerformQualityRampupExperiment() posts to the adaptation task queue to maybe reset video source restrictions on the Processor. |quality_rampup_done_| is made std::atomic. Bug: webrtc:11542, webrtc:11520 Change-Id: I1cfd76e0cd42f006a6d2527f5aa2aeb5266ba6d6 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/174441 Reviewed-by: Evan Shrubsole <eshr@google.com> Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org> Commit-Queue: Henrik Boström <hbos@webrtc.org> Cr-Commit-Position: refs/heads/master@{#31231}
2020-05-12 18:49:07 +02:00
RTC_DCHECK(resource_adaptation_queue_);
RTC_DCHECK_RUN_ON(resource_adaptation_queue_);
[Adaptation] Make ResourceUsageState nullable, remove kStable. This CL is part of the Call-Level Adaptation Processing design doc: https://docs.google.com/document/d/1ZyC26yOCknrrcYa839ZWLxD6o6Gig5A3lVTh4E41074/edit?usp=sharing The ResourceUsageState was written as: {kOveruse, kStable, kUnderuse}. The assumption was that if a resource neither wanted to adapt up or down it would report kStable. But with the addition of Resource::IsAdaptationUpAllowed() (prior CL) the notion of being "stable" was already captured outside of ResourceUsageState. Furthermore, kStable failed to capture what IsAdaptationUpAllowed() did not: whether we can go up depends on the resulting resolution or frame rate (restrictions_after). Perhaps we can go up a little, but not a lot. This CL also adds Resource::ClearUsageState(). After applying an adaptation, all usage states become invalidated (new measurements are needed to know if we are still over- or underusing). This was always the case, but prior to this CL this was not accurately reflected in the Resource::usage_state() in-between measurements. Bug: webrtc:11172 Change-Id: I140ff3114025b7732e530564690783e168d2509b Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/173088 Reviewed-by: Evan Shrubsole <eshr@google.com> Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org> Commit-Queue: Henrik Boström <hbos@webrtc.org> Cr-Commit-Position: refs/heads/master@{#31110}
2020-04-20 12:04:12 +02:00
usage_state_ = absl::nullopt;
[Overuse] Implement Resource and ResourceUsageListener. The Resource interface (previously a skeleton not used outside of testing) is updated to inform listeners of changes to resource usage. Debugging methods are removed (Name, UsageUnitsOfMeasurements, CurrentUsage). The interface is implemented by OveruseFrameDetectorResourceAdaptationModule's inner classes EncodeUsageResource and QualityScalerResource. The new ResourceUsageListener interface is implemented by OveruseFrameDetectorResourceAdaptationModule. In order to avoid adding AdaptationObserverInterface::AdaptReason to the ResourceUsageListener interface, the module figures out if the reason is "kCpu" or "kQuality" by looking which Resource object triggered OnResourceUsageStateMeasured(). These resources no longer need an explicit reference to OveruseFrameDetectorResourceAdaptationModule and could potentially be used by a different module. In this CL, AdaptationObserverInterface::AdaptDown()'s return value is still needed by QualityScaler. This is mirrored in the return value of ResourceUsageListener::OnResourceUsageStateMeasured(). A TODO is added to remove it and a comment explains how the current implementation seems to break the contract of the method (as was the case prior to this CL). Follow-up work include: - Move EncodeUsageResource and QualityScalerResource to separate files. - Make resources injectable, allowing fake resources in testing and removing OnResourceOveruseForTesting() methods. (Investigate adding the necessary input signals to the Resource interface or relevant sub-interfaces so that the module does not need to know which Resource implementation is used.) - And more! See whiteboard :) Bug: webrtc:11222 Change-Id: I0a46ace4a2e617874e3ee97e67e3a199fef420a2 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/168180 Commit-Queue: Henrik Boström <hbos@webrtc.org> Reviewed-by: Erik Språng <sprang@webrtc.org> Reviewed-by: Evan Shrubsole <eshr@google.com> Cr-Commit-Position: refs/heads/master@{#30469}
2020-02-06 12:49:57 +01:00
}
[Adaptation] Resource::IsAdaptationUpAllowed() for IsBitrateConstrained. This CL is part of the Call-Level Adaptation Processing design doc: https://docs.google.com/document/d/1ZyC26yOCknrrcYa839ZWLxD6o6Gig5A3lVTh4E41074/edit?usp=sharing The VideoStreamAdapter is currently responsible for aborting and not providing adaptations if we are bitrate constrained (kIsBitrateConstrained). Whether or not we are bitrate constrained is clearly a resource question and should be phrased as such. By moving this logic to Resource::IsAdaptationUpAllowed(), the VideoStreamAdapter can continue to be thread-agnostic when a future CL introduces a "processing queue", and the VideoStreamAdapter can be simplified: it returns Adaptations even if we are constrained (but we refuse to Apply them any resource rejects it). This CL adds new Resource classes as inner classes of ResourceAdaptationProcessor that take on the responsibility of kIsBitrateConstrained logic: PreventIncreaseResolutionDueToBitrateResource and PreventAdaptUpInBalancedResource. A third class, PreventAdaptUpDueToActiveCounts, also allows us to move adaptation-aborting logic. This piece of code appears to be about not adapting up if we’re already at the highest setting, which would be VideoStreamAdapter responsibility (covered by Adaptation::Status::kLimitReached), but it is actually more complicated than that: the active_counts_ care about "reason", so it is really about "is this resource type OK with you adapting up?". We should probably rewrite this code in the future, but for now it is moved to an inner class of ResourceAdaptationProcessor. Other misc changes: - ApplyDegradationPreference is moved to video_stream_adapter.[h/cc] and renamed "Filter". - OnResourceOveruse/Underuse now use Resource* as the reason instead of AdaptReason. In a future CL, the processor will be split into a "processor" part and a "video stream encoder resource manager" part. Only the manager needs to know about AdaptReason since this is only used for |active_counts_| and we want to get rid of it as much as possible as it is not future-proof. Bug: webrtc:11172 Change-Id: I2eba9ec3d717f7024c451aeb14635fe759551318 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/172930 Commit-Queue: Henrik Boström <hbos@webrtc.org> Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org> Reviewed-by: Evan Shrubsole <eshr@google.com> Cr-Commit-Position: refs/heads/master@{#31099}
2020-04-17 13:48:21 +02:00
bool Resource::IsAdaptationUpAllowed(
const VideoStreamInputState& input_state,
const VideoSourceRestrictions& restrictions_before,
const VideoSourceRestrictions& restrictions_after,
[Adaptation] Make Resources reference counted and add more DCHECKs. In a future CL, adaptation processing and stream encoder resource management will happen on different task queues. When this is the case, asynchronous tasks will be posted in both directions and some resources will have internal states used on multiple threads. This CL makes the Resource class reference counted in order to support posting tasks to a different threads without risk of use-after-free when a posted task is executed with a delay. This is preferred over WeakPtr strategies because WeakPtrs are single-threaded and preferred over raw pointer usage because the reference counted approach enables more compile-time and run-time assurance. This is also "future proof"; when resources can be injected through public APIs, ownership needs to be shared between libwebrtc and the application (e.g. Chrome). To reduce the risk of making mistakes in the future CL, sequence checkers and task queue DCHECKs are added as well as other DCHECKs to make sure things have been cleaned up before destruction, e.g: - Processor gets a sequence checker. It is entirely single-threaded. - Processor must not have any attached listeners or resources on destruction. - Resources must not have any listeners on destruction. - The Manager, EncodeUsageResource and QualityScalerResource DCHECKs they are running on the encoder queue. - TODOs are added illustrating where we want to add PostTasks in the future CL. Lastly, upon VideoStreamEncoder::Stop() we delete the ResourceAdaptationProcessor. Because the Processor is already used in posted tasks, some if statements are added to ensure the Processor is not used after destruction. Bug: webrtc:11542, webrtc:11520 Change-Id: Ibaa8a61d86d87a71f477d1075a117c28d9d2d285 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/174760 Commit-Queue: Henrik Boström <hbos@webrtc.org> Reviewed-by: Evan Shrubsole <eshr@google.com> Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org> Cr-Commit-Position: refs/heads/master@{#31217}
2020-05-11 16:29:22 +02:00
rtc::scoped_refptr<Resource> reason_resource) const {
[Adaptation] Resource::IsAdaptationUpAllowed() for IsBitrateConstrained. This CL is part of the Call-Level Adaptation Processing design doc: https://docs.google.com/document/d/1ZyC26yOCknrrcYa839ZWLxD6o6Gig5A3lVTh4E41074/edit?usp=sharing The VideoStreamAdapter is currently responsible for aborting and not providing adaptations if we are bitrate constrained (kIsBitrateConstrained). Whether or not we are bitrate constrained is clearly a resource question and should be phrased as such. By moving this logic to Resource::IsAdaptationUpAllowed(), the VideoStreamAdapter can continue to be thread-agnostic when a future CL introduces a "processing queue", and the VideoStreamAdapter can be simplified: it returns Adaptations even if we are constrained (but we refuse to Apply them any resource rejects it). This CL adds new Resource classes as inner classes of ResourceAdaptationProcessor that take on the responsibility of kIsBitrateConstrained logic: PreventIncreaseResolutionDueToBitrateResource and PreventAdaptUpInBalancedResource. A third class, PreventAdaptUpDueToActiveCounts, also allows us to move adaptation-aborting logic. This piece of code appears to be about not adapting up if we’re already at the highest setting, which would be VideoStreamAdapter responsibility (covered by Adaptation::Status::kLimitReached), but it is actually more complicated than that: the active_counts_ care about "reason", so it is really about "is this resource type OK with you adapting up?". We should probably rewrite this code in the future, but for now it is moved to an inner class of ResourceAdaptationProcessor. Other misc changes: - ApplyDegradationPreference is moved to video_stream_adapter.[h/cc] and renamed "Filter". - OnResourceOveruse/Underuse now use Resource* as the reason instead of AdaptReason. In a future CL, the processor will be split into a "processor" part and a "video stream encoder resource manager" part. Only the manager needs to know about AdaptReason since this is only used for |active_counts_| and we want to get rid of it as much as possible as it is not future-proof. Bug: webrtc:11172 Change-Id: I2eba9ec3d717f7024c451aeb14635fe759551318 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/172930 Commit-Queue: Henrik Boström <hbos@webrtc.org> Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org> Reviewed-by: Evan Shrubsole <eshr@google.com> Cr-Commit-Position: refs/heads/master@{#31099}
2020-04-17 13:48:21 +02:00
return true;
}
[Adaptation] Add OnAdaptationApplied(), remove ResourceListenerResponse. This CL is part of the Call-Level Adaptation Processing design doc: https://docs.google.com/document/d/1ZyC26yOCknrrcYa839ZWLxD6o6Gig5A3lVTh4E41074/edit?usp=sharing The ResourceListenerResponse was used to make the QualityScaler not clear QP samples and instead increase its frequency of checking for QP under certain circumstances, see enum description: https://webrtc.googlesource.com/src.git/+/c70b1028d47c1aee4892545190cd66e97d09cd55/call/adaptation/resource.h#33 Because the QualityScaler depends on whether and how adaptation happened it should listen to adaptation happening. This CL moves the logic that was previously in VideoStreamAdapter closer to the QualityScaler: QualityScalerResource::OnAdaptationApplied(). This would allow the VideoStreamAdapter to operate on a separate task queue in the future, with no dependencies on any stream-specific resources that might operate on other task queues. Bug: webrtc:11172, webrtc:11521 Change-Id: I07971a8a5fab5715f4ccb7d2c63f1b92bd47170f Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/173090 Commit-Queue: Henrik Boström <hbos@webrtc.org> Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org> Reviewed-by: Evan Shrubsole <eshr@google.com> Cr-Commit-Position: refs/heads/master@{#31143}
2020-04-28 12:24:33 +02:00
void Resource::OnAdaptationApplied(
const VideoStreamInputState& input_state,
const VideoSourceRestrictions& restrictions_before,
const VideoSourceRestrictions& restrictions_after,
[Adaptation] Make Resources reference counted and add more DCHECKs. In a future CL, adaptation processing and stream encoder resource management will happen on different task queues. When this is the case, asynchronous tasks will be posted in both directions and some resources will have internal states used on multiple threads. This CL makes the Resource class reference counted in order to support posting tasks to a different threads without risk of use-after-free when a posted task is executed with a delay. This is preferred over WeakPtr strategies because WeakPtrs are single-threaded and preferred over raw pointer usage because the reference counted approach enables more compile-time and run-time assurance. This is also "future proof"; when resources can be injected through public APIs, ownership needs to be shared between libwebrtc and the application (e.g. Chrome). To reduce the risk of making mistakes in the future CL, sequence checkers and task queue DCHECKs are added as well as other DCHECKs to make sure things have been cleaned up before destruction, e.g: - Processor gets a sequence checker. It is entirely single-threaded. - Processor must not have any attached listeners or resources on destruction. - Resources must not have any listeners on destruction. - The Manager, EncodeUsageResource and QualityScalerResource DCHECKs they are running on the encoder queue. - TODOs are added illustrating where we want to add PostTasks in the future CL. Lastly, upon VideoStreamEncoder::Stop() we delete the ResourceAdaptationProcessor. Because the Processor is already used in posted tasks, some if statements are added to ensure the Processor is not used after destruction. Bug: webrtc:11542, webrtc:11520 Change-Id: Ibaa8a61d86d87a71f477d1075a117c28d9d2d285 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/174760 Commit-Queue: Henrik Boström <hbos@webrtc.org> Reviewed-by: Evan Shrubsole <eshr@google.com> Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org> Cr-Commit-Position: refs/heads/master@{#31217}
2020-05-11 16:29:22 +02:00
rtc::scoped_refptr<Resource> reason_resource) {}
[Adaptation] Add OnAdaptationApplied(), remove ResourceListenerResponse. This CL is part of the Call-Level Adaptation Processing design doc: https://docs.google.com/document/d/1ZyC26yOCknrrcYa839ZWLxD6o6Gig5A3lVTh4E41074/edit?usp=sharing The ResourceListenerResponse was used to make the QualityScaler not clear QP samples and instead increase its frequency of checking for QP under certain circumstances, see enum description: https://webrtc.googlesource.com/src.git/+/c70b1028d47c1aee4892545190cd66e97d09cd55/call/adaptation/resource.h#33 Because the QualityScaler depends on whether and how adaptation happened it should listen to adaptation happening. This CL moves the logic that was previously in VideoStreamAdapter closer to the QualityScaler: QualityScalerResource::OnAdaptationApplied(). This would allow the VideoStreamAdapter to operate on a separate task queue in the future, with no dependencies on any stream-specific resources that might operate on other task queues. Bug: webrtc:11172, webrtc:11521 Change-Id: I07971a8a5fab5715f4ccb7d2c63f1b92bd47170f Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/173090 Commit-Queue: Henrik Boström <hbos@webrtc.org> Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org> Reviewed-by: Evan Shrubsole <eshr@google.com> Cr-Commit-Position: refs/heads/master@{#31143}
2020-04-28 12:24:33 +02:00
[Adaptation] Move adaptation logic to a separate task queue. This CL unblocks future Call-Level Mitigation strategies by moving the ResourceAdaptationProcessor to a separate task queue. This signifies a major milestone in the new resource adaptation architecture because with this CL the threading model is in place and moving the Processor to the Call and increasing its responsibilities is made possible. In this CL, we still have one Processor per VideoStreamEncoder and the VideoStreamEncoder is responsible for the creation and the destruction of its Processor and that Processor's task queue. But the PostTasks are in place and the decision-making is executed on a separate queue. This CL: - Moves ResourceAdaptationProcessor to an adaptation task queue. It continues to be entirely single-threaded, but now operates on a separate task queue. - Makes Resources thread-safe: Interaction with the Processor, i.e. OnResourceUsageStateMeasured() and IsAdaptationUpAllowed(), happens on the adaptation task queue. State updates are pushed from the encoder task queue with PostTasks. - QualityScalerResource operates on both task queues; the QP usage callbacks are invoked asynchronously. - The VideoStreamEncoderResourceManager operates on the encoder task queue with the following exceptions: 1) Its resources are accessible on any thread (using a mutex). This is OK because resources are reference counted and thread safe. This aids adding and removing resources to the Processor on the adaptation task queue. 2) |active_counts_| is moved to the adaptation task queue. This makes it possible for PreventAdaptUpDueToActiveCounts to run IsAdaptationUpAllowed() on the adaptation task queue. A side-effect of this is that some stats reporting now happen on the adaptation task queue, but that is OK because VideoStreamEncoderObserver is thread-safe. The Manager is updated to take the new threading model into account: - OnFrameDroppedDueToSize() posts to the adaptation task queue to invoke the Processor. - OnVideoSourceRestrictionsUpdated(), now invoked on the adaptation task queue, updates |active_counts_| synchronously but posts to the encoder task queue to update video source restrictions (which it only uses to calculate target frame rate). - MaybePerformQualityRampupExperiment() posts to the adaptation task queue to maybe reset video source restrictions on the Processor. |quality_rampup_done_| is made std::atomic. Bug: webrtc:11542, webrtc:11520 Change-Id: I1cfd76e0cd42f006a6d2527f5aa2aeb5266ba6d6 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/174441 Reviewed-by: Evan Shrubsole <eshr@google.com> Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org> Commit-Queue: Henrik Boström <hbos@webrtc.org> Cr-Commit-Position: refs/heads/master@{#31231}
2020-05-12 18:49:07 +02:00
rtc::TaskQueue* Resource::encoder_queue() const {
return encoder_queue_;
}
rtc::TaskQueue* Resource::resource_adaptation_queue() const {
return resource_adaptation_queue_;
}
[Adaptation] Add OnAdaptationApplied(), remove ResourceListenerResponse. This CL is part of the Call-Level Adaptation Processing design doc: https://docs.google.com/document/d/1ZyC26yOCknrrcYa839ZWLxD6o6Gig5A3lVTh4E41074/edit?usp=sharing The ResourceListenerResponse was used to make the QualityScaler not clear QP samples and instead increase its frequency of checking for QP under certain circumstances, see enum description: https://webrtc.googlesource.com/src.git/+/c70b1028d47c1aee4892545190cd66e97d09cd55/call/adaptation/resource.h#33 Because the QualityScaler depends on whether and how adaptation happened it should listen to adaptation happening. This CL moves the logic that was previously in VideoStreamAdapter closer to the QualityScaler: QualityScalerResource::OnAdaptationApplied(). This would allow the VideoStreamAdapter to operate on a separate task queue in the future, with no dependencies on any stream-specific resources that might operate on other task queues. Bug: webrtc:11172, webrtc:11521 Change-Id: I07971a8a5fab5715f4ccb7d2c63f1b92bd47170f Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/173090 Commit-Queue: Henrik Boström <hbos@webrtc.org> Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org> Reviewed-by: Evan Shrubsole <eshr@google.com> Cr-Commit-Position: refs/heads/master@{#31143}
2020-04-28 12:24:33 +02:00
void Resource::OnResourceUsageStateMeasured(ResourceUsageState usage_state) {
[Adaptation] Move adaptation logic to a separate task queue. This CL unblocks future Call-Level Mitigation strategies by moving the ResourceAdaptationProcessor to a separate task queue. This signifies a major milestone in the new resource adaptation architecture because with this CL the threading model is in place and moving the Processor to the Call and increasing its responsibilities is made possible. In this CL, we still have one Processor per VideoStreamEncoder and the VideoStreamEncoder is responsible for the creation and the destruction of its Processor and that Processor's task queue. But the PostTasks are in place and the decision-making is executed on a separate queue. This CL: - Moves ResourceAdaptationProcessor to an adaptation task queue. It continues to be entirely single-threaded, but now operates on a separate task queue. - Makes Resources thread-safe: Interaction with the Processor, i.e. OnResourceUsageStateMeasured() and IsAdaptationUpAllowed(), happens on the adaptation task queue. State updates are pushed from the encoder task queue with PostTasks. - QualityScalerResource operates on both task queues; the QP usage callbacks are invoked asynchronously. - The VideoStreamEncoderResourceManager operates on the encoder task queue with the following exceptions: 1) Its resources are accessible on any thread (using a mutex). This is OK because resources are reference counted and thread safe. This aids adding and removing resources to the Processor on the adaptation task queue. 2) |active_counts_| is moved to the adaptation task queue. This makes it possible for PreventAdaptUpDueToActiveCounts to run IsAdaptationUpAllowed() on the adaptation task queue. A side-effect of this is that some stats reporting now happen on the adaptation task queue, but that is OK because VideoStreamEncoderObserver is thread-safe. The Manager is updated to take the new threading model into account: - OnFrameDroppedDueToSize() posts to the adaptation task queue to invoke the Processor. - OnVideoSourceRestrictionsUpdated(), now invoked on the adaptation task queue, updates |active_counts_| synchronously but posts to the encoder task queue to update video source restrictions (which it only uses to calculate target frame rate). - MaybePerformQualityRampupExperiment() posts to the adaptation task queue to maybe reset video source restrictions on the Processor. |quality_rampup_done_| is made std::atomic. Bug: webrtc:11542, webrtc:11520 Change-Id: I1cfd76e0cd42f006a6d2527f5aa2aeb5266ba6d6 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/174441 Reviewed-by: Evan Shrubsole <eshr@google.com> Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org> Commit-Queue: Henrik Boström <hbos@webrtc.org> Cr-Commit-Position: refs/heads/master@{#31231}
2020-05-12 18:49:07 +02:00
RTC_DCHECK(resource_adaptation_queue_);
RTC_DCHECK_RUN_ON(resource_adaptation_queue_);
[Overuse] Implement Resource and ResourceUsageListener. The Resource interface (previously a skeleton not used outside of testing) is updated to inform listeners of changes to resource usage. Debugging methods are removed (Name, UsageUnitsOfMeasurements, CurrentUsage). The interface is implemented by OveruseFrameDetectorResourceAdaptationModule's inner classes EncodeUsageResource and QualityScalerResource. The new ResourceUsageListener interface is implemented by OveruseFrameDetectorResourceAdaptationModule. In order to avoid adding AdaptationObserverInterface::AdaptReason to the ResourceUsageListener interface, the module figures out if the reason is "kCpu" or "kQuality" by looking which Resource object triggered OnResourceUsageStateMeasured(). These resources no longer need an explicit reference to OveruseFrameDetectorResourceAdaptationModule and could potentially be used by a different module. In this CL, AdaptationObserverInterface::AdaptDown()'s return value is still needed by QualityScaler. This is mirrored in the return value of ResourceUsageListener::OnResourceUsageStateMeasured(). A TODO is added to remove it and a comment explains how the current implementation seems to break the contract of the method (as was the case prior to this CL). Follow-up work include: - Move EncodeUsageResource and QualityScalerResource to separate files. - Make resources injectable, allowing fake resources in testing and removing OnResourceOveruseForTesting() methods. (Investigate adding the necessary input signals to the Resource interface or relevant sub-interfaces so that the module does not need to know which Resource implementation is used.) - And more! See whiteboard :) Bug: webrtc:11222 Change-Id: I0a46ace4a2e617874e3ee97e67e3a199fef420a2 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/168180 Commit-Queue: Henrik Boström <hbos@webrtc.org> Reviewed-by: Erik Språng <sprang@webrtc.org> Reviewed-by: Evan Shrubsole <eshr@google.com> Cr-Commit-Position: refs/heads/master@{#30469}
2020-02-06 12:49:57 +01:00
usage_state_ = usage_state;
[Adaptation] Make ResourceUsageState nullable, remove kStable. This CL is part of the Call-Level Adaptation Processing design doc: https://docs.google.com/document/d/1ZyC26yOCknrrcYa839ZWLxD6o6Gig5A3lVTh4E41074/edit?usp=sharing The ResourceUsageState was written as: {kOveruse, kStable, kUnderuse}. The assumption was that if a resource neither wanted to adapt up or down it would report kStable. But with the addition of Resource::IsAdaptationUpAllowed() (prior CL) the notion of being "stable" was already captured outside of ResourceUsageState. Furthermore, kStable failed to capture what IsAdaptationUpAllowed() did not: whether we can go up depends on the resulting resolution or frame rate (restrictions_after). Perhaps we can go up a little, but not a lot. This CL also adds Resource::ClearUsageState(). After applying an adaptation, all usage states become invalidated (new measurements are needed to know if we are still over- or underusing). This was always the case, but prior to this CL this was not accurately reflected in the Resource::usage_state() in-between measurements. Bug: webrtc:11172 Change-Id: I140ff3114025b7732e530564690783e168d2509b Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/173088 Reviewed-by: Evan Shrubsole <eshr@google.com> Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org> Commit-Queue: Henrik Boström <hbos@webrtc.org> Cr-Commit-Position: refs/heads/master@{#31110}
2020-04-20 12:04:12 +02:00
if (!listener_)
[Adaptation] Add OnAdaptationApplied(), remove ResourceListenerResponse. This CL is part of the Call-Level Adaptation Processing design doc: https://docs.google.com/document/d/1ZyC26yOCknrrcYa839ZWLxD6o6Gig5A3lVTh4E41074/edit?usp=sharing The ResourceListenerResponse was used to make the QualityScaler not clear QP samples and instead increase its frequency of checking for QP under certain circumstances, see enum description: https://webrtc.googlesource.com/src.git/+/c70b1028d47c1aee4892545190cd66e97d09cd55/call/adaptation/resource.h#33 Because the QualityScaler depends on whether and how adaptation happened it should listen to adaptation happening. This CL moves the logic that was previously in VideoStreamAdapter closer to the QualityScaler: QualityScalerResource::OnAdaptationApplied(). This would allow the VideoStreamAdapter to operate on a separate task queue in the future, with no dependencies on any stream-specific resources that might operate on other task queues. Bug: webrtc:11172, webrtc:11521 Change-Id: I07971a8a5fab5715f4ccb7d2c63f1b92bd47170f Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/173090 Commit-Queue: Henrik Boström <hbos@webrtc.org> Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org> Reviewed-by: Evan Shrubsole <eshr@google.com> Cr-Commit-Position: refs/heads/master@{#31143}
2020-04-28 12:24:33 +02:00
return;
[Adaptation] Make Resources reference counted and add more DCHECKs. In a future CL, adaptation processing and stream encoder resource management will happen on different task queues. When this is the case, asynchronous tasks will be posted in both directions and some resources will have internal states used on multiple threads. This CL makes the Resource class reference counted in order to support posting tasks to a different threads without risk of use-after-free when a posted task is executed with a delay. This is preferred over WeakPtr strategies because WeakPtrs are single-threaded and preferred over raw pointer usage because the reference counted approach enables more compile-time and run-time assurance. This is also "future proof"; when resources can be injected through public APIs, ownership needs to be shared between libwebrtc and the application (e.g. Chrome). To reduce the risk of making mistakes in the future CL, sequence checkers and task queue DCHECKs are added as well as other DCHECKs to make sure things have been cleaned up before destruction, e.g: - Processor gets a sequence checker. It is entirely single-threaded. - Processor must not have any attached listeners or resources on destruction. - Resources must not have any listeners on destruction. - The Manager, EncodeUsageResource and QualityScalerResource DCHECKs they are running on the encoder queue. - TODOs are added illustrating where we want to add PostTasks in the future CL. Lastly, upon VideoStreamEncoder::Stop() we delete the ResourceAdaptationProcessor. Because the Processor is already used in posted tasks, some if statements are added to ensure the Processor is not used after destruction. Bug: webrtc:11542, webrtc:11520 Change-Id: Ibaa8a61d86d87a71f477d1075a117c28d9d2d285 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/174760 Commit-Queue: Henrik Boström <hbos@webrtc.org> Reviewed-by: Evan Shrubsole <eshr@google.com> Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org> Cr-Commit-Position: refs/heads/master@{#31217}
2020-05-11 16:29:22 +02:00
listener_->OnResourceUsageStateMeasured(this);
Initial version of ResourceAdaptationProcessor and friends. This CL adds Resource, ResourceConsumer, ResourceConsumerConfiguration and ResourceAdaptationProcessor and implements the algorithm outlined in https://docs.google.com/presentation/d/13jyqCWNpIa873iKT6yDuB5Q5ma-c0CvxBpX--0tCclY/edit?usp=sharing. Simply put, if any resource (such as "CPU") is overusing, the most expensive consumer (e.g. encoded stream) is adapted one step down. If all resources are underusing, the least expensive consumer is adapted one step up. The current resources, consumers and configurations are all fakes; this CL has no effect on the current adaptation algorithms used in practise, but it lays down the foundation for future work in this area. Bug: webrtc:11167, webrtc:11168, webrtc:11169 Change-Id: I4054ec7728a52a49e137eee6fa67fa27debd9254 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/161237 Commit-Queue: Henrik Boström <hbos@webrtc.org> Reviewed-by: Evan Shrubsole <eshr@google.com> Reviewed-by: Stefan Holmer <stefan@webrtc.org> Reviewed-by: Erik Språng <sprang@webrtc.org> Cr-Commit-Position: refs/heads/master@{#30053}
2019-12-10 14:14:09 +01:00
}
} // namespace webrtc
Reference in New Issue Copy Permalink