前言
从应用层到native层查看mediacodec的创建
后续再看Acodec到omx
查看代码重点关注log
Codec相关：MediaCodec\MediaCodecSource\Acodec\MPEG4Writer\omx
音频数据：AudioSource
视频数据：GraphicBufferSource
1 应用层
以camera代码为例，因为camera使用的是mediarecorder，而audiorecord只能录制pcm数据。
android\packages\apps\SnapdragonCamera\src\com\android\camera\captureModule.java
1.1 创建
mMediaRecorder = new MediaRecorder()
1.2 获取
mMediaRecorder.setAudioSource(MediaRecorder.AudioSource.MIC);
int audioEncoder = SettingTranslation.getAudioEncoder(mSettingsManager.getValue(SettingsManager.KEY_AUDIO_ENCODER));
1.3 设置VideoSource
mMediaRecorder.setVideoSource(MediaRecorder.VideoSource.SURFACE);
MediaRecorder: setVideoSource(2)
视频数据来源为surface(屏幕显示？)，非camera，存疑
1.4 设置输出模式
mMediaRecorder.setOutputFormat(mProfile.fileFormat);
MediaRecorder: setOutputFormat(2)
选择输出模式为mpeg4，目前看起来输出模式只有两种
默认为OUTPUT_FORMAT_THREE_GPP
存疑：为什么输出模式有两种，如何对应encoder，而编码方式中也有mpeg4
MediaRecorder官方文档导读
1.5 设置文件名
mMediaRecorder.setOutputFile(fileName);
1.6 设置编码方式
mMediaRecorder.setVideoEncoder(mVideoEncoder);
MediaRecorder: setVideoEncoder(2)
选择编码方式为h264
编码格式不同于输出格式，以音频为类比。
视频编码可以分为h264/mpeg4，音频编码为lpcm/aac
视频输出格式常见的是mp4/m4a(outputformat为mpeg4)，音频输出格式mp3/wav/amr
视频编码为h264（camera app默认设置）
07-26 22:23:32.636 3772 3772 V MediaRecorder: setVideoEncoder(2)
07-26 23:01:36.513 1129 3967 I OMXMaster: makeComponentInstance(OMX.qcom.video.encoder.avc) in android.hardwar process
视频编码为mpeg4
07-26 23:02:09.983 3772 3772 V MediaRecorder: setVideoEncoder(3)
07-20 23:30:40.325 972 1051 I OMXMaster: makeComponentInstance(OMX.qcom.video.encoder.mpeg4) in android.hardwar process
视频编码为h265
07-26 23:02:28.156 3772 3772 V MediaRecorder: setVideoEncoder(5)
07-26 23:02:28.158 1121 1216 V MPEG4Writer: initInternal
07-26 23:02:28.158 1121 1216 V MediaCodecList: matching ‘OMX.qcom.video.encoder.hevc’
07-26 23:02:28.158 1121 1216 V MediaCodecList: matching ‘c2.android.hevc.encoder’
07-26 23:02:28.158 1121 1216 V ACodec : Now uninitialized
07-26 23:02:28.159 1121 4879 V ACodec : onAllocateComponent
07-26 23:02:28.161 1121 4879 I OMXClient: IOmx service obtained
07-26 23:02:28.162 1129 4672 I OMXMaster: makeComponentInstance(OMX.qcom.video.encoder.hevc) in android.hardwar process
c2.android.hevc.encoder的size不符合
设置h265编码时可选两种编码器，但是最终选择了qcom的硬件编码
1 判断是否是软解码，依据编码器名称是否包含OMX.google.和c2.android.
ps：startsWithIgnoreCase表示不区分大小写
2 判断软解码
以soundrecorder为例，可以看到输出格式为audio/3gpp
以camera mpeg4为例，可以看到输出格式为video/mp4v-es，输入格式为video/raw
以camera h265为例，可以看到输出格式为video/hevc，输入格式为video/raw
2 获取video codec表——mediaprofile
java代码
上层传入encoder类型
根据video_encoder类型，获取编码信息，码率/帧率/宽/高的最大最小值
存疑：sProfiles哪里来的
来源于/vendor/etc/media_profiles_vendor.xml
目前看起来不支持vp8，参见/vendor/etc/media_profiles_vendor.xml
3 MediaRecorder: prepare
3.1 创建MediaCodecSource（重点看MediaCodec的log）
status_t MediaCodecSource::initEncoder() {
//1. 根据传入的mimi等信息，查找mediacodeclist中合适的codec名字存入Vector<AString> matchingCodecs中
MediaCodecList::findMatchingCodecs(outputMIME.c_str(), true /* encoder */,mFlags,&matchingCodecs);
//2. 遍历matchingCodecs，根据名字创建mediacodec
//ps:还有一种方式MediaCodec::CreateByType创建mediacodec，或者函数有的时候会用到
mEncoder = MediaCodec::CreateByComponentName(mCodecLooper, matchingCodecs[ix]);
//3. 打印输出格式
ALOGV("output format is '%s'", mOutputFormat->debugString(0).c_str());
//4. 配置mediacodec
err = mEncoder->configure();
//5. 启动mediacodec
err = mEncoder->start();
}
1 查找匹配的codec名字
输入mime，在MediaCodecList中查找合适的codec名字，存入matches中
mime --> MediaCodecList.matches.name
07-28 03:44:05.130 945 3290 D MediaCodecSource: youkai in MediaCodecSource::initEncoder
07-28 03:44:05.130 945 3290 V MediaCodecList: youkai in MediaCodecList::findMatchingCodecs
07-28 03:44:05.130 945 3290 V MediaCodecList: matching 'OMX.qcom.video.encoder.mpeg4'
07-28 03:44:05.130 945 3290 V MediaCodecList: matching 'c2.android.mpeg4.encoder'
07-28 03:44:05.130 945 3290 V MediaCodecList: matching 'OMX.google.mpeg4.encoder'
至此匹配到三个encoder
07-28 03:44:05.130 945 3290 V MediaCodecList: matching ‘OMX.qcom.video.encoder.mpeg4’
Media_codecs.xml
07-28 03:44:05.130 945 3290 V MediaCodecList: matching ‘c2.android.mpeg4.encoder’
07-28 03:44:05.130 945 3290 V MediaCodecList: matching ‘OMX.google.mpeg4.encoder’
Media_codecs_sw.xml (frameworks\av\media\libstagefright\data)
Size不符合淘汰
2 根据名字创建mediacodec
3 打出输出格式，这个参数是从app传来的
07-28 03:44:05.161 945 15206 V ACodec : [OMX.qcom.video.encoder.mpeg4] Now Loaded
07-28 03:44:05.163 945 3290 V MediaCodecSource: output format is 'AMessage(what = 0x00000000) = {
07-28 03:44:05.163 945 3290 V MediaCodecSource: string mime = "video/mp4v-es"
07-28 03:44:05.163 945 3290 V MediaCodecSource: int32_t width = 1920
07-28 03:44:05.163 945 3290 V MediaCodecSource: int32_t height = 1080
07-28 03:44:05.163 945 3290 V MediaCodecSource: int32_t stride = 1920
07-28 03:44:05.163 945 3290 V MediaCodecSource: int32_t slice-height = 1080
07-28 03:44:05.163 945 3290 V MediaCodecSource: int32_t color-format = 2130708361
07-28 03:44:05.163 945 3290 V MediaCodecSource: int32_t bitrate = 20000000
07-28 03:44:05.163 945 3290 V MediaCodecSource: int32_t frame-rate = 30
07-28 03:44:05.163 945 3290 V MediaCodecSource: int32_t i-frame-interval = 1
07-28 03:44:05.163 945 3290 V MediaCodecSource: int32_t priority = 0
07-28 03:44:05.163 945 3290 V MediaCodecSource: int32_t feature-nal-length-bitstream = 1
07-28 03:44:05.163 945 3290 V MediaCodecSource: int32_t nal-length-in-bytes = 4
07-28 03:44:05.163 945 3290 V MediaCodecSource: int32_t isNativeRecorder = 1
07-28 03:44:05.163 945 3290 V MediaCodecSource: int32_t create-input-buffers-suspended = 1
07-28 03:44:05.163 945 3290 V MediaCodecSource: }'
07-28 03:44:05.163 945 15205 I MediaCodec: MediaCodec will operate in async mode
07-28 03:44:05.163 945 15205 V MediaCodec: kWhatConfigure: Old mCrypto: 0x0 (0)
07-28 03:44:05.163 945 15205 V MediaCodec: kWhatConfigure: New mCrypto: 0x0 (0)
07-28 03:44:05.163 945 15205 V MediaCodec: Found 0 pieces of codec specific data.
07-28 03:44:05.163 945 15206 V ACodec : onConfigureComponent
4 mediacodec的消息机制接受到kWhatComponentConfigured，打印出输入和输出格式
07-28 03:44:05.226 945 15205 V MediaCodec: [OMX.qcom.video.encoder.mpeg4] configured as input format: AMessage(what = 0x00000000) = {
07-28 03:44:05.226 945 15205 V MediaCodec: string mime = "video/raw"
07-28 03:44:05.226 945 15205 V MediaCodec: int32_t stride = 1920
07-28 03:44:05.226 945 15205 V MediaCodec: int32_t slice-height = 1080
07-28 03:44:05.226 945 15205 V MediaCodec: int32_t color-format = 2130708361
07-28 03:44:05.226 945 15205 V MediaCodec: int32_t color-range = 0
07-28 03:44:05.226 945 15205 V MediaCodec: int32_t color-standard = 0
07-28 03:44:05.226 945 15205 V MediaCodec: int32_t color-transfer = 0
07-28 03:44:05.226 945 15205 V MediaCodec: int32_t width = 1920
07-28 03:44:05.226 945 15205 V MediaCodec: int32_t height = 1080
07-28 03:44:05.226 945 15205 V MediaCodec: }, output format: AMessage(what = 0x00000000) = {
07-28 03:44:05.226 945 15205 V MediaCodec: int32_t bitrate = 20000000
07-28 03:44:05.226 945 15205 V MediaCodec: int32_t max-bitrate = 20000000
07-28 03:44:05.226 945 15205 V MediaCodec: int32_t frame-rate = 30
07-28 03:44:05.226 945 15205 V MediaCodec: string mime = "video/mp4v-es"
07-28 03:44:05.226 945 15205 V MediaCodec: int32_t width = 1920
07-28 03:44:05.226 945 15205 V MediaCodec: int32_t height = 1080
07-28 03:44:05.226 945 15205 V MediaCodec: }
07-28 03:44:05.227 945 15206 V ACodec : onCreateInputSurface
07-28 03:44:05.227 953 2087 V GraphicBufferSource: GraphicBufferSource
5 接受到GraphicBufferSource传来的创建显示消息kWhatInputSurfaceCreated
07-28 03:44:05.232 945 15205 V MediaCodec: [OMX.qcom.video.encoder.mpeg4] input surface created as input format: AMessage(what = 0x00000000) = {
07-28 03:44:05.232 945 15205 V MediaCodec: string mime = "video/raw"
07-28 03:44:05.232 945 15205 V MediaCodec: int32_t stride = 1920
07-28 03:44:05.232 945 15205 V MediaCodec: int32_t slice-height = 1080
07-28 03:44:05.232 945 15205 V MediaCodec: int32_t color-format = 2130708361
07-28 03:44:05.232 945 15205 V MediaCodec: int32_t color-range = 2
07-28 03:44:05.232 945 15205 V MediaCodec: int32_t color-standard = 1
07-28 03:44:05.232 945 15205 V MediaCodec: int32_t color-transfer = 3
07-28 03:44:05.232 945 15205 V MediaCodec: int32_t width = 1920
07-28 03:44:05.232 945 15205 V MediaCodec: int32_t height = 1080
07-28 03:44:05.232 945 15205 V MediaCodec: int32_t android._dataspace = 260
07-28 03:44:05.232 945 15205 V MediaCodec: Buffer android._color-aspects = {
07-28 03:44:05.232 945 15205 V MediaCodec: 00000000: 02 00 00 00 01 00 00 00 03 00 00 00 01 00 00 00 ................
07-28 03:44:05.232 945 15205 V MediaCodec: }
07-28 03:44:05.232 945 15205 V MediaCodec: }, output format: AMessage(what = 0x00000000) = {
07-28 03:44:05.232 945 15205 V MediaCodec: int32_t bitrate = 20000000
07-28 03:44:05.232 945 15205 V MediaCodec: int32_t max-bitrate = 20000000
07-28 03:44:05.232 945 15205 V MediaCodec: int32_t frame-rate = 30
07-28 03:44:05.232 945 15205 V MediaCodec: string mime = "video/mp4v-es"
07-28 03:44:05.232 945 15205 V MediaCodec: int32_t width = 1920
07-28 03:44:05.232 945 15205 V MediaCodec: int32_t height = 1080
07-28 03:44:05.232 945 15205 V MediaCodec: int32_t color-range = 2
07-28 03:44:05.232 945 15205 V MediaCodec: int32_t color-standard = 1
07-28 03:44:05.232 945 15205 V MediaCodec: int32_t color-transfer = 3
07-28 03:44:05.232 945 15205 V MediaCodec: }
07-28 03:44:05.232 945 3290 V MediaCodecSource: setting dataspace 0x104, format 0x22
07-28 03:44:05.232 945 15206 V ACodec : onStart
07-28 03:44:05.232 953 1026 V GraphicBufferSource: stop
3.2 Audio
07-28 03:44:05.303 945 3290 D MediaCodecSource: youkai in MediaCodecSource::initEncoder
07-28 03:44:05.303 945 3290 V ACodec : Now uninitialized
07-28 03:44:05.304 945 15212 V ACodec : onAllocateComponent
07-28 03:44:05.306 945 15212 I OMXClient: IOmx service obtained
07-28 03:44:05.306 714 15211 V AudioFlinger: acquireWakeLock_l() AudioIn_7E status 0
07-28 03:44:05.306 714 15211 D AudioFlinger: updateWakeLockUids_l AudioIn_7E uids:
07-28 03:44:05.306 953 1026 I OMXMaster: makeComponentInstance(OMX.qcom.audio.encoder.aac) in android.hardwar process
07-28 03:44:05.306 714 15211 V AudioFlinger: updateWakeLockUids_l() AudioIn_7E status 0
07-28 03:44:05.308 714 15211 V AudioFlinger: releaseWakeLock_l() AudioIn_7E
07-28 03:44:05.309 714 15211 V AudioFlinger: RecordThread: loop stopping
07-28 03:44:05.314 953 1026 E : component init: role = OMX.qcom.audio.encoder.aac
07-28 03:44:05.324 0 0 I aac_in_open: session id 2: NT mode encoder success
07-28 03:44:05.324 0 0 I aac_in_open: session id 2: success
07-28 03:44:05.327 953 1026 W media.codec: Failed to obtain quirks for omx component 'OMX.qcom.audio.encoder.aac' from XML files
07-28 03:44:05.328 945 15212 V ACodec : [OMX.qcom.audio.encoder.aac] Now Loaded
07-28 03:44:05.329 945 3290 V MediaCodecSource: output format is 'AMessage(what = 0x00000000) = {
07-28 03:44:05.329 945 3290 V MediaCodecSource: string mime = "audio/mp4a-latm"
07-28 03:44:05.329 945 3290 V MediaCodecSource: int32_t aac-profile = 2
07-28 03:44:05.329 945 3290 V MediaCodecSource: int32_t max-input-size = 2048
07-28 03:44:05.329 945 3290 V MediaCodecSource: int32_t channel-count = 2
07-28 03:44:05.329 945 3290 V MediaCodecSource: int32_t sample-rate = 48000
07-28 03:44:05.329 945 3290 V MediaCodecSource: int32_t bitrate = 96000
07-28 03:44:05.329 945 3290 V MediaCodecSource: int32_t priority = 0
07-28 03:44:05.329 945 3290 V MediaCodecSource: }'
07-28 03:44:05.330 945 15212 I MediaCodec: MediaCodec will operate in async mode
07-28 03:44:05.331 945 15212 V MediaCodec: kWhatConfigure: Old mCrypto: 0x0 (0)
07-28 03:44:05.332 945 15212 V MediaCodec: kWhatConfigure: New mCrypto: 0x0 (0)
07-28 03:44:05.332 945 15212 V MediaCodec: Found 0 pieces of codec specific data.
07-28 03:44:05.332 945 15212 V ACodec : onConfigureComponent
07-28 03:44:05.334 945 15212 V MediaCodec: [OMX.qcom.audio.encoder.aac] configured as input format: AMessage(what = 0x00000000) = {
07-28 03:44:05.334 945 15212 V MediaCodec: string mime = "audio/raw"
07-28 03:44:05.334 945 15212 V MediaCodec: int32_t channel-count = 2
07-28 03:44:05.334 945 15212 V MediaCodec: int32_t sample-rate = 48000
07-28 03:44:05.334 945 15212 V MediaCodec: int32_t pcm-encoding = 2
07-28 03:44:05.334 945 15212 V MediaCodec: }, output format: AMessage(what = 0x00000000) = {
07-28 03:44:05.334 945 15212 V MediaCodec: int32_t bitrate = 96000
07-28 03:44:05.334 945 15212 V MediaCodec: int32_t max-bitrate = 96000
07-28 03:44:05.334 945 15212 V MediaCodec: string mime = "audio/mp4a-latm"
07-28 03:44:05.334 945 15212 V MediaCodec: int32_t channel-count = 2
07-28 03:44:05.334 945 15212 V MediaCodec: int32_t sample-rate = 48000
07-28 03:44:05.334 945 15212 V MediaCodec: }
07-28 03:44:05.335 945 3290 V MediaCodecSource: setting dataspace 0x10c10000, format 0x22
07-28 03:44:05.335 945 15212 V ACodec : onStart
07-28 03:44:05.367 945 15212 V MediaCodec: [OMX.qcom.audio.encoder.aac] output format changed to: AMessage(what = 0x00000000) = {
07-28 03:44:05.367 945 15212 V MediaCodec: int32_t bitrate = 96000
07-28 03:44:05.367 945 15212 V MediaCodec: int32_t max-bitrate = 96000
07-28 03:44:05.367 945 15212 V MediaCodec: string mime = "audio/mp4a-latm"
07-28 03:44:05.367 945 15212 V MediaCodec: int32_t channel-count = 2
07-28 03:44:05.367 945 15212 V MediaCodec: int32_t sample-rate = 48000
07-28 03:44:05.367 945 15212 V MediaCodec: }
07-28 03:44:05.367 945 15212 V MediaCodec: -- returned buffer timestamp 0 <= 0, ignore it
07-28 03:44:05.367 945 15204 E Utils : csd0 too small
07-28 03:44:05.367 945 15204 E ExtendedUtils: csd0 too small
07-28 03:44:05.368 945 15212 V ACodec : [OMX.qcom.audio.encoder.aac] calling fillBuffer 3
4 MediaRecorder: start
07-28 03:44:05.756 14402 14402 V MediaRecorder: start
07-28 03:44:05.756 945 1028 V StagefrightRecorder: start
07-28 03:44:05.756 945 1028 V MPEG4Writer: movie time scale: 1000
07-28 03:44:05.756 945 1028 V MPEG4Writer: muxer starting: mHasMoovBox 1, mHasFileLevelMeta 0
07-28 03:44:05.757 945 1028 I MPEG4Writer: limits: 11065696640/600000000 bytes/us, bit rate: 20096000 bps and the estimated moov size 405120 bytes
07-28 03:44:05.757 945 1028 V MPEG4Writer: startWriterThread
07-28 03:44:05.757 945 1028 V MPEG4Writer: initTrackingProgressStatus
07-28 03:44:05.757 945 1028 I MPEG4Writer: Start time offset: 100000 us
07-28 03:44:05.757 945 15239 V MPEG4Writer: ThreadWrapper: 0xeb153180
07-28 03:44:05.757 945 15239 V MPEG4Writer: threadFunc
07-28 03:44:05.757 945 15239 V MPEG4Writer: findChunkToWrite
07-28 03:44:05.757 945 15239 V MPEG4Writer: Nothing to be written after all
07-28 03:44:05.757 945 15204 I MediaCodecSource: MediaCodecSource (video) starting
07-28 03:44:05.758 945 15204 I MediaCodecSource: MediaCodecSource (video) started
07-28 03:44:05.758 945 1028 V MPEG4Writer: initTrackingProgressStatus
07-28 03:44:05.759 945 1028 I MPEG4Writer: Start time offset: 100000 us
07-28 03:44:05.759 945 15204 I MediaCodecSource: MediaCodecSource (audio) starting
07-28 03:44:05.759 945 15204 V MediaCodecSource: puller (audio) start
07-28 03:44:05.759 953 14820 D GraphicBufferSource: setStartTimeUs: skipFramesBeforeUs=81188109321
07-28 03:44:05.760 953 1026 V GraphicBufferSource: setSuspend=0 at time -1 us
07-28 03:44:05.763 945 15241 V AudioRecord: start(48): sync event 0 trigger session 0
5 小结
本章主要介绍从java的mediarecorder到mediacodec创建流程。
- 从app传下来结果关键参数(编码类型)
- 经过mediaprofile的过滤，选择出支持的格式
- 将mime传给mediacodeclist选择合适的encoder名字
- 根据encoder名字创建mediacodec(这里会创建Acodec，然后到omx和MPEG4writer，后续文章介绍)
mediarecorder的prepare阶段重点创建音视频的encoder，start阶段简单介绍