webrtc_m130/webrtc/audio/audio_transport_proxy.cc

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

138 lines
5.7 KiB
C++
Raw Normal View History

/*
* Copyright (c) 2016 The WebRTC project authors. All Rights Reserved.
*
* Use of this source code is governed by a BSD-style license
* that can be found in the LICENSE file in the root of the source
* tree. An additional intellectual property rights grant can be found
* in the file PATENTS. All contributing project authors may
* be found in the AUTHORS file in the root of the source tree.
*/
#include "webrtc/audio/audio_transport_proxy.h"
namespace webrtc {
Replace AudioConferenceMixer with AudioMixer. This CL re-routes audio through AudioMixer instead of AudioConferenceMixer. This is done without any modifications to VoiceEngine. Previously, output audio was polled by an AudioDevice through an AudioTransport pointer, which was an instance of VoEBaseImpl. VoiceEngineImpl sent the request for data on to OutputMixer and further to AudioConferenceMixer. This CL changes the audio flow to an AudioDevice. We reconfigure the AudioDevice to have another AudioTransport pointer, which points to an AudioTransportProxy. The AudioTransportProxy is responsible for feeding mixed data to the AudioProcessing component for echo cancellation, and to resample the audio data after AudioProcessing and before it is sent to the AudioDevice. The set up of the audio path was previously done during VoiceEngine initialization. Now it is changed in the AudioState constructor. This list shows where audio-path-related VoiceEngine functionality has been moved: OutputMixer --> AudioTransportProxy VoiceEngineImpl --> AudioState, AudioTransportProxy SharedData --> AudioState Channel --> AudioReceiveStream, ChannelProxy, Channel AudioState owns the new mixer and connects it to AudioTransport and AudioDevice on initialization. The audio input source is AudioReceiveStream, which registers itself with the mixer (which it gets from AudioState) on Start and Stop. # Since the AudioTransport interface contains non-const references. NOPRESUBMIT=True BUG=webrtc:6346 Review-Url: https://codereview.webrtc.org/2436033002 Cr-Commit-Position: refs/heads/master@{#15193}
2016-11-22 06:42:53 -08:00
namespace {
// Resample audio in |frame| to given sample rate preserving the
// channel count and place the result in |destination|.
int Resample(const AudioFrame& frame,
const int destination_sample_rate,
PushResampler<int16_t>* resampler,
int16_t* destination) {
const int number_of_channels = static_cast<int>(frame.num_channels_);
const int target_number_of_samples_per_channel =
destination_sample_rate / 100;
resampler->InitializeIfNeeded(frame.sample_rate_hz_, destination_sample_rate,
number_of_channels);
// TODO(yujo): make resampler take an AudioFrame, and add special case
// handling of muted frames.
Replace AudioConferenceMixer with AudioMixer. This CL re-routes audio through AudioMixer instead of AudioConferenceMixer. This is done without any modifications to VoiceEngine. Previously, output audio was polled by an AudioDevice through an AudioTransport pointer, which was an instance of VoEBaseImpl. VoiceEngineImpl sent the request for data on to OutputMixer and further to AudioConferenceMixer. This CL changes the audio flow to an AudioDevice. We reconfigure the AudioDevice to have another AudioTransport pointer, which points to an AudioTransportProxy. The AudioTransportProxy is responsible for feeding mixed data to the AudioProcessing component for echo cancellation, and to resample the audio data after AudioProcessing and before it is sent to the AudioDevice. The set up of the audio path was previously done during VoiceEngine initialization. Now it is changed in the AudioState constructor. This list shows where audio-path-related VoiceEngine functionality has been moved: OutputMixer --> AudioTransportProxy VoiceEngineImpl --> AudioState, AudioTransportProxy SharedData --> AudioState Channel --> AudioReceiveStream, ChannelProxy, Channel AudioState owns the new mixer and connects it to AudioTransport and AudioDevice on initialization. The audio input source is AudioReceiveStream, which registers itself with the mixer (which it gets from AudioState) on Start and Stop. # Since the AudioTransport interface contains non-const references. NOPRESUBMIT=True BUG=webrtc:6346 Review-Url: https://codereview.webrtc.org/2436033002 Cr-Commit-Position: refs/heads/master@{#15193}
2016-11-22 06:42:53 -08:00
return resampler->Resample(
frame.data(), frame.samples_per_channel_ * number_of_channels,
destination, number_of_channels * target_number_of_samples_per_channel);
Replace AudioConferenceMixer with AudioMixer. This CL re-routes audio through AudioMixer instead of AudioConferenceMixer. This is done without any modifications to VoiceEngine. Previously, output audio was polled by an AudioDevice through an AudioTransport pointer, which was an instance of VoEBaseImpl. VoiceEngineImpl sent the request for data on to OutputMixer and further to AudioConferenceMixer. This CL changes the audio flow to an AudioDevice. We reconfigure the AudioDevice to have another AudioTransport pointer, which points to an AudioTransportProxy. The AudioTransportProxy is responsible for feeding mixed data to the AudioProcessing component for echo cancellation, and to resample the audio data after AudioProcessing and before it is sent to the AudioDevice. The set up of the audio path was previously done during VoiceEngine initialization. Now it is changed in the AudioState constructor. This list shows where audio-path-related VoiceEngine functionality has been moved: OutputMixer --> AudioTransportProxy VoiceEngineImpl --> AudioState, AudioTransportProxy SharedData --> AudioState Channel --> AudioReceiveStream, ChannelProxy, Channel AudioState owns the new mixer and connects it to AudioTransport and AudioDevice on initialization. The audio input source is AudioReceiveStream, which registers itself with the mixer (which it gets from AudioState) on Start and Stop. # Since the AudioTransport interface contains non-const references. NOPRESUBMIT=True BUG=webrtc:6346 Review-Url: https://codereview.webrtc.org/2436033002 Cr-Commit-Position: refs/heads/master@{#15193}
2016-11-22 06:42:53 -08:00
}
} // namespace
AudioTransportProxy::AudioTransportProxy(AudioTransport* voe_audio_transport,
AudioProcessing* audio_processing,
AudioMixer* mixer)
: voe_audio_transport_(voe_audio_transport),
audio_processing_(audio_processing),
mixer_(mixer) {
RTC_DCHECK(voe_audio_transport);
RTC_DCHECK(audio_processing);
Replace AudioConferenceMixer with AudioMixer. This CL re-routes audio through AudioMixer instead of AudioConferenceMixer. This is done without any modifications to VoiceEngine. Previously, output audio was polled by an AudioDevice through an AudioTransport pointer, which was an instance of VoEBaseImpl. VoiceEngineImpl sent the request for data on to OutputMixer and further to AudioConferenceMixer. This CL changes the audio flow to an AudioDevice. We reconfigure the AudioDevice to have another AudioTransport pointer, which points to an AudioTransportProxy. The AudioTransportProxy is responsible for feeding mixed data to the AudioProcessing component for echo cancellation, and to resample the audio data after AudioProcessing and before it is sent to the AudioDevice. The set up of the audio path was previously done during VoiceEngine initialization. Now it is changed in the AudioState constructor. This list shows where audio-path-related VoiceEngine functionality has been moved: OutputMixer --> AudioTransportProxy VoiceEngineImpl --> AudioState, AudioTransportProxy SharedData --> AudioState Channel --> AudioReceiveStream, ChannelProxy, Channel AudioState owns the new mixer and connects it to AudioTransport and AudioDevice on initialization. The audio input source is AudioReceiveStream, which registers itself with the mixer (which it gets from AudioState) on Start and Stop. # Since the AudioTransport interface contains non-const references. NOPRESUBMIT=True BUG=webrtc:6346 Review-Url: https://codereview.webrtc.org/2436033002 Cr-Commit-Position: refs/heads/master@{#15193}
2016-11-22 06:42:53 -08:00
RTC_DCHECK(mixer);
}
AudioTransportProxy::~AudioTransportProxy() {}
int32_t AudioTransportProxy::RecordedDataIsAvailable(
const void* audioSamples,
const size_t nSamples,
const size_t nBytesPerSample,
const size_t nChannels,
const uint32_t samplesPerSec,
const uint32_t totalDelayMS,
const int32_t clockDrift,
const uint32_t currentMicLevel,
const bool keyPressed,
Reland of Enable cpplint and fix cpplint errors in webrtc/*audio (patchset #1 id:1 of https://codereview.webrtc.org/2739143002/ ) Reason for revert: Can reland it if backwards compatible API is kept. Original issue's description: > Revert of Enable cpplint and fix cpplint errors in webrtc/*audio (patchset #4 id:180001 of https://codereview.webrtc.org/2683033004/ ) > > Reason for revert: > The API change in audio/utility/audio_frame_operations.h caused breakage. Need to keep backward-compatible API. > > Original issue's description: > > Enable cpplint and fix cpplint errors in webrtc/*audio > > > > Change usage accordingly throughout the codebase > > > > BUG=webrtc:5268 > > > > TESTED=Fixed issues reported by: > > find webrtc/*audio -type f -name *.cc -o -name *.h | xargs cpplint.py > > > > Review-Url: https://codereview.webrtc.org/2683033004 > > Cr-Commit-Position: refs/heads/master@{#17133} > > Committed: https://chromium.googlesource.com/external/webrtc/+/aebe55ca6c2f725c81031722badb297a2fb1d331 > > TBR=henrika@webrtc.org,henrik.lundin@webrtc.org,kwiberg@webrtc.org > # Skipping CQ checks because original CL landed less than 1 days ago. > NOPRESUBMIT=true > NOTREECHECKS=true > NOTRY=true > BUG=webrtc:5268 > > Review-Url: https://codereview.webrtc.org/2739143002 > Cr-Commit-Position: refs/heads/master@{#17138} > Committed: https://chromium.googlesource.com/external/webrtc/+/e47c1d3ca172e5ea84e59c1e88dbe25664664e54 TBR=henrika@webrtc.org,henrik.lundin@webrtc.org,kwiberg@webrtc.org # Skipping CQ checks because original CL landed less than 1 days ago. BUG=webrtc:5268 Review-Url: https://codereview.webrtc.org/2739073003 Cr-Commit-Position: refs/heads/master@{#17144}
2017-03-09 06:25:06 -08:00
uint32_t& newMicLevel) { // NOLINT: to avoid changing APIs
// Pass call through to original audio transport instance.
return voe_audio_transport_->RecordedDataIsAvailable(
audioSamples, nSamples, nBytesPerSample, nChannels, samplesPerSec,
totalDelayMS, clockDrift, currentMicLevel, keyPressed, newMicLevel);
}
int32_t AudioTransportProxy::NeedMorePlayData(const size_t nSamples,
const size_t nBytesPerSample,
const size_t nChannels,
const uint32_t samplesPerSec,
void* audioSamples,
size_t& nSamplesOut,
int64_t* elapsed_time_ms,
int64_t* ntp_time_ms) {
RTC_DCHECK_EQ(sizeof(int16_t) * nChannels, nBytesPerSample);
RTC_DCHECK_GE(nChannels, 1);
RTC_DCHECK_LE(nChannels, 2);
RTC_DCHECK_GE(
samplesPerSec,
static_cast<uint32_t>(AudioProcessing::NativeRate::kSampleRate8kHz));
Replace AudioConferenceMixer with AudioMixer. This CL re-routes audio through AudioMixer instead of AudioConferenceMixer. This is done without any modifications to VoiceEngine. Previously, output audio was polled by an AudioDevice through an AudioTransport pointer, which was an instance of VoEBaseImpl. VoiceEngineImpl sent the request for data on to OutputMixer and further to AudioConferenceMixer. This CL changes the audio flow to an AudioDevice. We reconfigure the AudioDevice to have another AudioTransport pointer, which points to an AudioTransportProxy. The AudioTransportProxy is responsible for feeding mixed data to the AudioProcessing component for echo cancellation, and to resample the audio data after AudioProcessing and before it is sent to the AudioDevice. The set up of the audio path was previously done during VoiceEngine initialization. Now it is changed in the AudioState constructor. This list shows where audio-path-related VoiceEngine functionality has been moved: OutputMixer --> AudioTransportProxy VoiceEngineImpl --> AudioState, AudioTransportProxy SharedData --> AudioState Channel --> AudioReceiveStream, ChannelProxy, Channel AudioState owns the new mixer and connects it to AudioTransport and AudioDevice on initialization. The audio input source is AudioReceiveStream, which registers itself with the mixer (which it gets from AudioState) on Start and Stop. # Since the AudioTransport interface contains non-const references. NOPRESUBMIT=True BUG=webrtc:6346 Review-Url: https://codereview.webrtc.org/2436033002 Cr-Commit-Position: refs/heads/master@{#15193}
2016-11-22 06:42:53 -08:00
// 100 = 1 second / data duration (10 ms).
RTC_DCHECK_EQ(nSamples * 100, samplesPerSec);
RTC_DCHECK_LE(nBytesPerSample * nSamples * nChannels,
AudioFrame::kMaxDataSizeBytes);
Replace AudioConferenceMixer with AudioMixer. This CL re-routes audio through AudioMixer instead of AudioConferenceMixer. This is done without any modifications to VoiceEngine. Previously, output audio was polled by an AudioDevice through an AudioTransport pointer, which was an instance of VoEBaseImpl. VoiceEngineImpl sent the request for data on to OutputMixer and further to AudioConferenceMixer. This CL changes the audio flow to an AudioDevice. We reconfigure the AudioDevice to have another AudioTransport pointer, which points to an AudioTransportProxy. The AudioTransportProxy is responsible for feeding mixed data to the AudioProcessing component for echo cancellation, and to resample the audio data after AudioProcessing and before it is sent to the AudioDevice. The set up of the audio path was previously done during VoiceEngine initialization. Now it is changed in the AudioState constructor. This list shows where audio-path-related VoiceEngine functionality has been moved: OutputMixer --> AudioTransportProxy VoiceEngineImpl --> AudioState, AudioTransportProxy SharedData --> AudioState Channel --> AudioReceiveStream, ChannelProxy, Channel AudioState owns the new mixer and connects it to AudioTransport and AudioDevice on initialization. The audio input source is AudioReceiveStream, which registers itself with the mixer (which it gets from AudioState) on Start and Stop. # Since the AudioTransport interface contains non-const references. NOPRESUBMIT=True BUG=webrtc:6346 Review-Url: https://codereview.webrtc.org/2436033002 Cr-Commit-Position: refs/heads/master@{#15193}
2016-11-22 06:42:53 -08:00
mixer_->Mix(nChannels, &mixed_frame_);
*elapsed_time_ms = mixed_frame_.elapsed_time_ms_;
*ntp_time_ms = mixed_frame_.ntp_time_ms_;
const auto error = audio_processing_->ProcessReverseStream(&mixed_frame_);
Replace AudioConferenceMixer with AudioMixer. This CL re-routes audio through AudioMixer instead of AudioConferenceMixer. This is done without any modifications to VoiceEngine. Previously, output audio was polled by an AudioDevice through an AudioTransport pointer, which was an instance of VoEBaseImpl. VoiceEngineImpl sent the request for data on to OutputMixer and further to AudioConferenceMixer. This CL changes the audio flow to an AudioDevice. We reconfigure the AudioDevice to have another AudioTransport pointer, which points to an AudioTransportProxy. The AudioTransportProxy is responsible for feeding mixed data to the AudioProcessing component for echo cancellation, and to resample the audio data after AudioProcessing and before it is sent to the AudioDevice. The set up of the audio path was previously done during VoiceEngine initialization. Now it is changed in the AudioState constructor. This list shows where audio-path-related VoiceEngine functionality has been moved: OutputMixer --> AudioTransportProxy VoiceEngineImpl --> AudioState, AudioTransportProxy SharedData --> AudioState Channel --> AudioReceiveStream, ChannelProxy, Channel AudioState owns the new mixer and connects it to AudioTransport and AudioDevice on initialization. The audio input source is AudioReceiveStream, which registers itself with the mixer (which it gets from AudioState) on Start and Stop. # Since the AudioTransport interface contains non-const references. NOPRESUBMIT=True BUG=webrtc:6346 Review-Url: https://codereview.webrtc.org/2436033002 Cr-Commit-Position: refs/heads/master@{#15193}
2016-11-22 06:42:53 -08:00
RTC_DCHECK_EQ(error, AudioProcessing::kNoError);
nSamplesOut = Resample(mixed_frame_, samplesPerSec, &resampler_,
static_cast<int16_t*>(audioSamples));
RTC_DCHECK_EQ(nSamplesOut, nChannels * nSamples);
return 0;
}
void AudioTransportProxy::PushCaptureData(int voe_channel,
const void* audio_data,
int bits_per_sample,
int sample_rate,
size_t number_of_channels,
size_t number_of_frames) {
// This is part of deprecated VoE interface operating on specific
// VoE channels. It should not be used.
RTC_NOTREACHED();
}
void AudioTransportProxy::PullRenderData(int bits_per_sample,
int sample_rate,
size_t number_of_channels,
size_t number_of_frames,
void* audio_data,
int64_t* elapsed_time_ms,
int64_t* ntp_time_ms) {
RTC_DCHECK_EQ(bits_per_sample, 16);
RTC_DCHECK_GE(number_of_channels, 1);
RTC_DCHECK_LE(number_of_channels, 2);
RTC_DCHECK_GE(sample_rate, AudioProcessing::NativeRate::kSampleRate8kHz);
Replace AudioConferenceMixer with AudioMixer. This CL re-routes audio through AudioMixer instead of AudioConferenceMixer. This is done without any modifications to VoiceEngine. Previously, output audio was polled by an AudioDevice through an AudioTransport pointer, which was an instance of VoEBaseImpl. VoiceEngineImpl sent the request for data on to OutputMixer and further to AudioConferenceMixer. This CL changes the audio flow to an AudioDevice. We reconfigure the AudioDevice to have another AudioTransport pointer, which points to an AudioTransportProxy. The AudioTransportProxy is responsible for feeding mixed data to the AudioProcessing component for echo cancellation, and to resample the audio data after AudioProcessing and before it is sent to the AudioDevice. The set up of the audio path was previously done during VoiceEngine initialization. Now it is changed in the AudioState constructor. This list shows where audio-path-related VoiceEngine functionality has been moved: OutputMixer --> AudioTransportProxy VoiceEngineImpl --> AudioState, AudioTransportProxy SharedData --> AudioState Channel --> AudioReceiveStream, ChannelProxy, Channel AudioState owns the new mixer and connects it to AudioTransport and AudioDevice on initialization. The audio input source is AudioReceiveStream, which registers itself with the mixer (which it gets from AudioState) on Start and Stop. # Since the AudioTransport interface contains non-const references. NOPRESUBMIT=True BUG=webrtc:6346 Review-Url: https://codereview.webrtc.org/2436033002 Cr-Commit-Position: refs/heads/master@{#15193}
2016-11-22 06:42:53 -08:00
// 100 = 1 second / data duration (10 ms).
RTC_DCHECK_EQ(number_of_frames * 100, sample_rate);
Replace AudioConferenceMixer with AudioMixer. This CL re-routes audio through AudioMixer instead of AudioConferenceMixer. This is done without any modifications to VoiceEngine. Previously, output audio was polled by an AudioDevice through an AudioTransport pointer, which was an instance of VoEBaseImpl. VoiceEngineImpl sent the request for data on to OutputMixer and further to AudioConferenceMixer. This CL changes the audio flow to an AudioDevice. We reconfigure the AudioDevice to have another AudioTransport pointer, which points to an AudioTransportProxy. The AudioTransportProxy is responsible for feeding mixed data to the AudioProcessing component for echo cancellation, and to resample the audio data after AudioProcessing and before it is sent to the AudioDevice. The set up of the audio path was previously done during VoiceEngine initialization. Now it is changed in the AudioState constructor. This list shows where audio-path-related VoiceEngine functionality has been moved: OutputMixer --> AudioTransportProxy VoiceEngineImpl --> AudioState, AudioTransportProxy SharedData --> AudioState Channel --> AudioReceiveStream, ChannelProxy, Channel AudioState owns the new mixer and connects it to AudioTransport and AudioDevice on initialization. The audio input source is AudioReceiveStream, which registers itself with the mixer (which it gets from AudioState) on Start and Stop. # Since the AudioTransport interface contains non-const references. NOPRESUBMIT=True BUG=webrtc:6346 Review-Url: https://codereview.webrtc.org/2436033002 Cr-Commit-Position: refs/heads/master@{#15193}
2016-11-22 06:42:53 -08:00
// 8 = bits per byte.
RTC_DCHECK_LE(bits_per_sample / 8 * number_of_frames * number_of_channels,
AudioFrame::kMaxDataSizeBytes);
Replace AudioConferenceMixer with AudioMixer. This CL re-routes audio through AudioMixer instead of AudioConferenceMixer. This is done without any modifications to VoiceEngine. Previously, output audio was polled by an AudioDevice through an AudioTransport pointer, which was an instance of VoEBaseImpl. VoiceEngineImpl sent the request for data on to OutputMixer and further to AudioConferenceMixer. This CL changes the audio flow to an AudioDevice. We reconfigure the AudioDevice to have another AudioTransport pointer, which points to an AudioTransportProxy. The AudioTransportProxy is responsible for feeding mixed data to the AudioProcessing component for echo cancellation, and to resample the audio data after AudioProcessing and before it is sent to the AudioDevice. The set up of the audio path was previously done during VoiceEngine initialization. Now it is changed in the AudioState constructor. This list shows where audio-path-related VoiceEngine functionality has been moved: OutputMixer --> AudioTransportProxy VoiceEngineImpl --> AudioState, AudioTransportProxy SharedData --> AudioState Channel --> AudioReceiveStream, ChannelProxy, Channel AudioState owns the new mixer and connects it to AudioTransport and AudioDevice on initialization. The audio input source is AudioReceiveStream, which registers itself with the mixer (which it gets from AudioState) on Start and Stop. # Since the AudioTransport interface contains non-const references. NOPRESUBMIT=True BUG=webrtc:6346 Review-Url: https://codereview.webrtc.org/2436033002 Cr-Commit-Position: refs/heads/master@{#15193}
2016-11-22 06:42:53 -08:00
mixer_->Mix(number_of_channels, &mixed_frame_);
*elapsed_time_ms = mixed_frame_.elapsed_time_ms_;
*ntp_time_ms = mixed_frame_.ntp_time_ms_;
const auto output_samples = Resample(mixed_frame_, sample_rate, &resampler_,
static_cast<int16_t*>(audio_data));
RTC_DCHECK_EQ(output_samples, number_of_channels * number_of_frames);
}
} // namespace webrtc