Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

82 lines
3.0 KiB
C
Raw Normal View History

/*
* Copyright 2019 The WebRTC Project Authors. All rights reserved.
*
* Use of this source code is governed by a BSD-style license
* that can be found in the LICENSE file in the root of the source
* tree. An additional intellectual property rights grant can be found
* in the file PATENTS. All contributing project authors may
* be found in the AUTHORS file in the root of the source tree.
*/
#ifndef CALL_ADAPTATION_RESOURCE_H_
#define CALL_ADAPTATION_RESOURCE_H_
#include <string>
[Overuse] Implement Resource and ResourceUsageListener. The Resource interface (previously a skeleton not used outside of testing) is updated to inform listeners of changes to resource usage. Debugging methods are removed (Name, UsageUnitsOfMeasurements, CurrentUsage). The interface is implemented by OveruseFrameDetectorResourceAdaptationModule's inner classes EncodeUsageResource and QualityScalerResource. The new ResourceUsageListener interface is implemented by OveruseFrameDetectorResourceAdaptationModule. In order to avoid adding AdaptationObserverInterface::AdaptReason to the ResourceUsageListener interface, the module figures out if the reason is "kCpu" or "kQuality" by looking which Resource object triggered OnResourceUsageStateMeasured(). These resources no longer need an explicit reference to OveruseFrameDetectorResourceAdaptationModule and could potentially be used by a different module. In this CL, AdaptationObserverInterface::AdaptDown()'s return value is still needed by QualityScaler. This is mirrored in the return value of ResourceUsageListener::OnResourceUsageStateMeasured(). A TODO is added to remove it and a comment explains how the current implementation seems to break the contract of the method (as was the case prior to this CL). Follow-up work include: - Move EncodeUsageResource and QualityScalerResource to separate files. - Make resources injectable, allowing fake resources in testing and removing OnResourceOveruseForTesting() methods. (Investigate adding the necessary input signals to the Resource interface or relevant sub-interfaces so that the module does not need to know which Resource implementation is used.) - And more! See whiteboard :) Bug: webrtc:11222 Change-Id: I0a46ace4a2e617874e3ee97e67e3a199fef420a2 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/168180 Commit-Queue: Henrik Boström <hbos@webrtc.org> Reviewed-by: Erik Språng <sprang@webrtc.org> Reviewed-by: Evan Shrubsole <eshr@google.com> Cr-Commit-Position: refs/heads/master@{#30469}
2020-02-06 12:49:57 +01:00
#include <vector>
#include "absl/types/optional.h"
[Adaptation] Make Resources reference counted and add more DCHECKs. In a future CL, adaptation processing and stream encoder resource management will happen on different task queues. When this is the case, asynchronous tasks will be posted in both directions and some resources will have internal states used on multiple threads. This CL makes the Resource class reference counted in order to support posting tasks to a different threads without risk of use-after-free when a posted task is executed with a delay. This is preferred over WeakPtr strategies because WeakPtrs are single-threaded and preferred over raw pointer usage because the reference counted approach enables more compile-time and run-time assurance. This is also "future proof"; when resources can be injected through public APIs, ownership needs to be shared between libwebrtc and the application (e.g. Chrome). To reduce the risk of making mistakes in the future CL, sequence checkers and task queue DCHECKs are added as well as other DCHECKs to make sure things have been cleaned up before destruction, e.g: - Processor gets a sequence checker. It is entirely single-threaded. - Processor must not have any attached listeners or resources on destruction. - Resources must not have any listeners on destruction. - The Manager, EncodeUsageResource and QualityScalerResource DCHECKs they are running on the encoder queue. - TODOs are added illustrating where we want to add PostTasks in the future CL. Lastly, upon VideoStreamEncoder::Stop() we delete the ResourceAdaptationProcessor. Because the Processor is already used in posted tasks, some if statements are added to ensure the Processor is not used after destruction. Bug: webrtc:11542, webrtc:11520 Change-Id: Ibaa8a61d86d87a71f477d1075a117c28d9d2d285 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/174760 Commit-Queue: Henrik Boström <hbos@webrtc.org> Reviewed-by: Evan Shrubsole <eshr@google.com> Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org> Cr-Commit-Position: refs/heads/master@{#31217}
2020-05-11 16:29:22 +02:00
#include "api/scoped_refptr.h"
#include "api/task_queue/task_queue_base.h"
[Adaptation] Resource::IsAdaptationUpAllowed() for IsBitrateConstrained. This CL is part of the Call-Level Adaptation Processing design doc: https://docs.google.com/document/d/1ZyC26yOCknrrcYa839ZWLxD6o6Gig5A3lVTh4E41074/edit?usp=sharing The VideoStreamAdapter is currently responsible for aborting and not providing adaptations if we are bitrate constrained (kIsBitrateConstrained). Whether or not we are bitrate constrained is clearly a resource question and should be phrased as such. By moving this logic to Resource::IsAdaptationUpAllowed(), the VideoStreamAdapter can continue to be thread-agnostic when a future CL introduces a "processing queue", and the VideoStreamAdapter can be simplified: it returns Adaptations even if we are constrained (but we refuse to Apply them any resource rejects it). This CL adds new Resource classes as inner classes of ResourceAdaptationProcessor that take on the responsibility of kIsBitrateConstrained logic: PreventIncreaseResolutionDueToBitrateResource and PreventAdaptUpInBalancedResource. A third class, PreventAdaptUpDueToActiveCounts, also allows us to move adaptation-aborting logic. This piece of code appears to be about not adapting up if we’re already at the highest setting, which would be VideoStreamAdapter responsibility (covered by Adaptation::Status::kLimitReached), but it is actually more complicated than that: the active_counts_ care about "reason", so it is really about "is this resource type OK with you adapting up?". We should probably rewrite this code in the future, but for now it is moved to an inner class of ResourceAdaptationProcessor. Other misc changes: - ApplyDegradationPreference is moved to video_stream_adapter.[h/cc] and renamed "Filter". - OnResourceOveruse/Underuse now use Resource* as the reason instead of AdaptReason. In a future CL, the processor will be split into a "processor" part and a "video stream encoder resource manager" part. Only the manager needs to know about AdaptReason since this is only used for |active_counts_| and we want to get rid of it as much as possible as it is not future-proof. Bug: webrtc:11172 Change-Id: I2eba9ec3d717f7024c451aeb14635fe759551318 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/172930 Commit-Queue: Henrik Boström <hbos@webrtc.org> Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org> Reviewed-by: Evan Shrubsole <eshr@google.com> Cr-Commit-Position: refs/heads/master@{#31099}
2020-04-17 13:48:21 +02:00
#include "call/adaptation/video_source_restrictions.h"
#include "call/adaptation/video_stream_input_state.h"
[Adaptation] Make Resources reference counted and add more DCHECKs. In a future CL, adaptation processing and stream encoder resource management will happen on different task queues. When this is the case, asynchronous tasks will be posted in both directions and some resources will have internal states used on multiple threads. This CL makes the Resource class reference counted in order to support posting tasks to a different threads without risk of use-after-free when a posted task is executed with a delay. This is preferred over WeakPtr strategies because WeakPtrs are single-threaded and preferred over raw pointer usage because the reference counted approach enables more compile-time and run-time assurance. This is also "future proof"; when resources can be injected through public APIs, ownership needs to be shared between libwebrtc and the application (e.g. Chrome). To reduce the risk of making mistakes in the future CL, sequence checkers and task queue DCHECKs are added as well as other DCHECKs to make sure things have been cleaned up before destruction, e.g: - Processor gets a sequence checker. It is entirely single-threaded. - Processor must not have any attached listeners or resources on destruction. - Resources must not have any listeners on destruction. - The Manager, EncodeUsageResource and QualityScalerResource DCHECKs they are running on the encoder queue. - TODOs are added illustrating where we want to add PostTasks in the future CL. Lastly, upon VideoStreamEncoder::Stop() we delete the ResourceAdaptationProcessor. Because the Processor is already used in posted tasks, some if statements are added to ensure the Processor is not used after destruction. Bug: webrtc:11542, webrtc:11520 Change-Id: Ibaa8a61d86d87a71f477d1075a117c28d9d2d285 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/174760 Commit-Queue: Henrik Boström <hbos@webrtc.org> Reviewed-by: Evan Shrubsole <eshr@google.com> Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org> Cr-Commit-Position: refs/heads/master@{#31217}
2020-05-11 16:29:22 +02:00
#include "rtc_base/ref_count.h"
namespace webrtc {
[Overuse] Implement Resource and ResourceUsageListener. The Resource interface (previously a skeleton not used outside of testing) is updated to inform listeners of changes to resource usage. Debugging methods are removed (Name, UsageUnitsOfMeasurements, CurrentUsage). The interface is implemented by OveruseFrameDetectorResourceAdaptationModule's inner classes EncodeUsageResource and QualityScalerResource. The new ResourceUsageListener interface is implemented by OveruseFrameDetectorResourceAdaptationModule. In order to avoid adding AdaptationObserverInterface::AdaptReason to the ResourceUsageListener interface, the module figures out if the reason is "kCpu" or "kQuality" by looking which Resource object triggered OnResourceUsageStateMeasured(). These resources no longer need an explicit reference to OveruseFrameDetectorResourceAdaptationModule and could potentially be used by a different module. In this CL, AdaptationObserverInterface::AdaptDown()'s return value is still needed by QualityScaler. This is mirrored in the return value of ResourceUsageListener::OnResourceUsageStateMeasured(). A TODO is added to remove it and a comment explains how the current implementation seems to break the contract of the method (as was the case prior to this CL). Follow-up work include: - Move EncodeUsageResource and QualityScalerResource to separate files. - Make resources injectable, allowing fake resources in testing and removing OnResourceOveruseForTesting() methods. (Investigate adding the necessary input signals to the Resource interface or relevant sub-interfaces so that the module does not need to know which Resource implementation is used.) - And more! See whiteboard :) Bug: webrtc:11222 Change-Id: I0a46ace4a2e617874e3ee97e67e3a199fef420a2 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/168180 Commit-Queue: Henrik Boström <hbos@webrtc.org> Reviewed-by: Erik Språng <sprang@webrtc.org> Reviewed-by: Evan Shrubsole <eshr@google.com> Cr-Commit-Position: refs/heads/master@{#30469}
2020-02-06 12:49:57 +01:00
class Resource;
enum class ResourceUsageState {
// Action is needed to minimze the load on this resource.
kOveruse,
// Increasing the load on this resource is desired, if possible.
kUnderuse,
};
const char* ResourceUsageStateToString(ResourceUsageState usage_state);
[Overuse] Implement Resource and ResourceUsageListener. The Resource interface (previously a skeleton not used outside of testing) is updated to inform listeners of changes to resource usage. Debugging methods are removed (Name, UsageUnitsOfMeasurements, CurrentUsage). The interface is implemented by OveruseFrameDetectorResourceAdaptationModule's inner classes EncodeUsageResource and QualityScalerResource. The new ResourceUsageListener interface is implemented by OveruseFrameDetectorResourceAdaptationModule. In order to avoid adding AdaptationObserverInterface::AdaptReason to the ResourceUsageListener interface, the module figures out if the reason is "kCpu" or "kQuality" by looking which Resource object triggered OnResourceUsageStateMeasured(). These resources no longer need an explicit reference to OveruseFrameDetectorResourceAdaptationModule and could potentially be used by a different module. In this CL, AdaptationObserverInterface::AdaptDown()'s return value is still needed by QualityScaler. This is mirrored in the return value of ResourceUsageListener::OnResourceUsageStateMeasured(). A TODO is added to remove it and a comment explains how the current implementation seems to break the contract of the method (as was the case prior to this CL). Follow-up work include: - Move EncodeUsageResource and QualityScalerResource to separate files. - Make resources injectable, allowing fake resources in testing and removing OnResourceOveruseForTesting() methods. (Investigate adding the necessary input signals to the Resource interface or relevant sub-interfaces so that the module does not need to know which Resource implementation is used.) - And more! See whiteboard :) Bug: webrtc:11222 Change-Id: I0a46ace4a2e617874e3ee97e67e3a199fef420a2 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/168180 Commit-Queue: Henrik Boström <hbos@webrtc.org> Reviewed-by: Erik Språng <sprang@webrtc.org> Reviewed-by: Evan Shrubsole <eshr@google.com> Cr-Commit-Position: refs/heads/master@{#30469}
2020-02-06 12:49:57 +01:00
class ResourceListener {
public:
virtual ~ResourceListener();
// Informs the listener of a new measurement of resource usage. This means
[Adaptation] Make Resources reference counted and add more DCHECKs. In a future CL, adaptation processing and stream encoder resource management will happen on different task queues. When this is the case, asynchronous tasks will be posted in both directions and some resources will have internal states used on multiple threads. This CL makes the Resource class reference counted in order to support posting tasks to a different threads without risk of use-after-free when a posted task is executed with a delay. This is preferred over WeakPtr strategies because WeakPtrs are single-threaded and preferred over raw pointer usage because the reference counted approach enables more compile-time and run-time assurance. This is also "future proof"; when resources can be injected through public APIs, ownership needs to be shared between libwebrtc and the application (e.g. Chrome). To reduce the risk of making mistakes in the future CL, sequence checkers and task queue DCHECKs are added as well as other DCHECKs to make sure things have been cleaned up before destruction, e.g: - Processor gets a sequence checker. It is entirely single-threaded. - Processor must not have any attached listeners or resources on destruction. - Resources must not have any listeners on destruction. - The Manager, EncodeUsageResource and QualityScalerResource DCHECKs they are running on the encoder queue. - TODOs are added illustrating where we want to add PostTasks in the future CL. Lastly, upon VideoStreamEncoder::Stop() we delete the ResourceAdaptationProcessor. Because the Processor is already used in posted tasks, some if statements are added to ensure the Processor is not used after destruction. Bug: webrtc:11542, webrtc:11520 Change-Id: Ibaa8a61d86d87a71f477d1075a117c28d9d2d285 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/174760 Commit-Queue: Henrik Boström <hbos@webrtc.org> Reviewed-by: Evan Shrubsole <eshr@google.com> Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org> Cr-Commit-Position: refs/heads/master@{#31217}
2020-05-11 16:29:22 +02:00
// that |resource->usage_state()| is now up-to-date.
virtual void OnResourceUsageStateMeasured(
rtc::scoped_refptr<Resource> resource) = 0;
[Overuse] Implement Resource and ResourceUsageListener. The Resource interface (previously a skeleton not used outside of testing) is updated to inform listeners of changes to resource usage. Debugging methods are removed (Name, UsageUnitsOfMeasurements, CurrentUsage). The interface is implemented by OveruseFrameDetectorResourceAdaptationModule's inner classes EncodeUsageResource and QualityScalerResource. The new ResourceUsageListener interface is implemented by OveruseFrameDetectorResourceAdaptationModule. In order to avoid adding AdaptationObserverInterface::AdaptReason to the ResourceUsageListener interface, the module figures out if the reason is "kCpu" or "kQuality" by looking which Resource object triggered OnResourceUsageStateMeasured(). These resources no longer need an explicit reference to OveruseFrameDetectorResourceAdaptationModule and could potentially be used by a different module. In this CL, AdaptationObserverInterface::AdaptDown()'s return value is still needed by QualityScaler. This is mirrored in the return value of ResourceUsageListener::OnResourceUsageStateMeasured(). A TODO is added to remove it and a comment explains how the current implementation seems to break the contract of the method (as was the case prior to this CL). Follow-up work include: - Move EncodeUsageResource and QualityScalerResource to separate files. - Make resources injectable, allowing fake resources in testing and removing OnResourceOveruseForTesting() methods. (Investigate adding the necessary input signals to the Resource interface or relevant sub-interfaces so that the module does not need to know which Resource implementation is used.) - And more! See whiteboard :) Bug: webrtc:11222 Change-Id: I0a46ace4a2e617874e3ee97e67e3a199fef420a2 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/168180 Commit-Queue: Henrik Boström <hbos@webrtc.org> Reviewed-by: Erik Språng <sprang@webrtc.org> Reviewed-by: Evan Shrubsole <eshr@google.com> Cr-Commit-Position: refs/heads/master@{#30469}
2020-02-06 12:49:57 +01:00
};
// A Resource monitors an implementation-specific system resource. It may report
// kOveruse or kUnderuse when resource usage is high or low enough that we
// should perform some sort of mitigation to fulfil the resource's constraints.
//
// All methods defined in this interface, except SetResourceListener(), MUST be
// invoked on the resource adaptation task queue.
//
// Usage measurements may be performed on an implementation-specific task queue.
// The Resource is reference counted to prevent use-after-free when posting
// between task queues. As such, the implementation MUST NOT make any
// assumptions about which task queue Resource is destructed on.
[Adaptation] Make Resources reference counted and add more DCHECKs. In a future CL, adaptation processing and stream encoder resource management will happen on different task queues. When this is the case, asynchronous tasks will be posted in both directions and some resources will have internal states used on multiple threads. This CL makes the Resource class reference counted in order to support posting tasks to a different threads without risk of use-after-free when a posted task is executed with a delay. This is preferred over WeakPtr strategies because WeakPtrs are single-threaded and preferred over raw pointer usage because the reference counted approach enables more compile-time and run-time assurance. This is also "future proof"; when resources can be injected through public APIs, ownership needs to be shared between libwebrtc and the application (e.g. Chrome). To reduce the risk of making mistakes in the future CL, sequence checkers and task queue DCHECKs are added as well as other DCHECKs to make sure things have been cleaned up before destruction, e.g: - Processor gets a sequence checker. It is entirely single-threaded. - Processor must not have any attached listeners or resources on destruction. - Resources must not have any listeners on destruction. - The Manager, EncodeUsageResource and QualityScalerResource DCHECKs they are running on the encoder queue. - TODOs are added illustrating where we want to add PostTasks in the future CL. Lastly, upon VideoStreamEncoder::Stop() we delete the ResourceAdaptationProcessor. Because the Processor is already used in posted tasks, some if statements are added to ensure the Processor is not used after destruction. Bug: webrtc:11542, webrtc:11520 Change-Id: Ibaa8a61d86d87a71f477d1075a117c28d9d2d285 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/174760 Commit-Queue: Henrik Boström <hbos@webrtc.org> Reviewed-by: Evan Shrubsole <eshr@google.com> Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org> Cr-Commit-Position: refs/heads/master@{#31217}
2020-05-11 16:29:22 +02:00
class Resource : public rtc::RefCountInterface {
public:
[Overuse] Implement Resource and ResourceUsageListener. The Resource interface (previously a skeleton not used outside of testing) is updated to inform listeners of changes to resource usage. Debugging methods are removed (Name, UsageUnitsOfMeasurements, CurrentUsage). The interface is implemented by OveruseFrameDetectorResourceAdaptationModule's inner classes EncodeUsageResource and QualityScalerResource. The new ResourceUsageListener interface is implemented by OveruseFrameDetectorResourceAdaptationModule. In order to avoid adding AdaptationObserverInterface::AdaptReason to the ResourceUsageListener interface, the module figures out if the reason is "kCpu" or "kQuality" by looking which Resource object triggered OnResourceUsageStateMeasured(). These resources no longer need an explicit reference to OveruseFrameDetectorResourceAdaptationModule and could potentially be used by a different module. In this CL, AdaptationObserverInterface::AdaptDown()'s return value is still needed by QualityScaler. This is mirrored in the return value of ResourceUsageListener::OnResourceUsageStateMeasured(). A TODO is added to remove it and a comment explains how the current implementation seems to break the contract of the method (as was the case prior to this CL). Follow-up work include: - Move EncodeUsageResource and QualityScalerResource to separate files. - Make resources injectable, allowing fake resources in testing and removing OnResourceOveruseForTesting() methods. (Investigate adding the necessary input signals to the Resource interface or relevant sub-interfaces so that the module does not need to know which Resource implementation is used.) - And more! See whiteboard :) Bug: webrtc:11222 Change-Id: I0a46ace4a2e617874e3ee97e67e3a199fef420a2 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/168180 Commit-Queue: Henrik Boström <hbos@webrtc.org> Reviewed-by: Erik Språng <sprang@webrtc.org> Reviewed-by: Evan Shrubsole <eshr@google.com> Cr-Commit-Position: refs/heads/master@{#30469}
2020-02-06 12:49:57 +01:00
Resource();
// Destruction may happen on any task queue.
[Adaptation] Make Resources reference counted and add more DCHECKs. In a future CL, adaptation processing and stream encoder resource management will happen on different task queues. When this is the case, asynchronous tasks will be posted in both directions and some resources will have internal states used on multiple threads. This CL makes the Resource class reference counted in order to support posting tasks to a different threads without risk of use-after-free when a posted task is executed with a delay. This is preferred over WeakPtr strategies because WeakPtrs are single-threaded and preferred over raw pointer usage because the reference counted approach enables more compile-time and run-time assurance. This is also "future proof"; when resources can be injected through public APIs, ownership needs to be shared between libwebrtc and the application (e.g. Chrome). To reduce the risk of making mistakes in the future CL, sequence checkers and task queue DCHECKs are added as well as other DCHECKs to make sure things have been cleaned up before destruction, e.g: - Processor gets a sequence checker. It is entirely single-threaded. - Processor must not have any attached listeners or resources on destruction. - Resources must not have any listeners on destruction. - The Manager, EncodeUsageResource and QualityScalerResource DCHECKs they are running on the encoder queue. - TODOs are added illustrating where we want to add PostTasks in the future CL. Lastly, upon VideoStreamEncoder::Stop() we delete the ResourceAdaptationProcessor. Because the Processor is already used in posted tasks, some if statements are added to ensure the Processor is not used after destruction. Bug: webrtc:11542, webrtc:11520 Change-Id: Ibaa8a61d86d87a71f477d1075a117c28d9d2d285 Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/174760 Commit-Queue: Henrik Boström <hbos@webrtc.org> Reviewed-by: Evan Shrubsole <eshr@google.com> Reviewed-by: Ilya Nikolaevskiy <ilnik@webrtc.org> Cr-Commit-Position: refs/heads/master@{#31217}
2020-05-11 16:29:22 +02:00
~Resource() override;
virtual std::string Name() const = 0;
// The listener MUST be informed any time UsageState() changes.
virtual void SetResourceListener(ResourceListener* listener) = 0;
// Within a single task running on the adaptation task queue, UsageState()
// MUST return the same value every time it is called.
// TODO(https://crbug.com/webrtc/11618): Remove the UsageState() getter in
// favor of passing the use usage state directly to the ResourceListener. This
// gets rid of this strange requirement of having to return the same thing
// every time.
virtual absl::optional<ResourceUsageState> UsageState() const = 0;
// Invalidates current usage measurements, i.e. in response to the system load
// changing. Example: an adaptation was just applied.
virtual void ClearUsageState() = 0;
};
} // namespace webrtc
#endif // CALL_ADAPTATION_RESOURCE_H_