Update: crash analysis, a working workaround (no reflash) and everything I ruled out
Short version: on my Vollaphone 22 the PulseAudio crash at the start of a call is a deterministic out-of-bounds write inside the Android audio HAL, caused by an invalid shared-memory header. Resetting the mixer control Speech_SHM_Init (plus five stuck switches) and restarting PulseAudio brought calls back without reflashing. I do not yet fully understand why the broken state survives reboots, so please read the workaround as “works on one device, not yet stress-tested”.
TL;DR
- Symptom: the call connects, then there is no audio on either side. PulseAudio segfaults right after
PCM_2_EN event 0x1 (call setup), restarts, and segfaults again.
- Crash site:
android::sph_memcpy() in audio.primary.mt6768.so, called from SpeechExtMemCCCI::shm_region_write_from_linear(). It writes to base + 0xfffffdff instead of somewhere inside a 0xd000 byte shared-memory mapping (/dev/ccci_raw_audio).
- Why: the region descriptor in that shared memory contains
offset = 0xfffffdff, size = 0xffffffff instead of 0x80 / 0x3000. The HAL only checks for size == 0, so it happily writes into nowhere.
- Workaround that worked: set
Speech_SHM_Init and the five stuck switches to 0 and restart PulseAudio. A small systemd unit now does the Speech_SHM_Init part at every boot.
- Not known yet: what exactly triggers the recovery (see “Open questions”), and whether the unit survives a hard reset.
Setup
- Vollaphone 22 (MediaTek MT6768), kernel
4.19.191-volla-mimameid-h12, Droidian phosh image (api32) flashed with flash_all.sh on 1 September
- PulseAudio loads
/android/vendor/lib64/hw/audio.primary.mt6768.so directly via libhybris, Android runs in lxc@android, callaudiod requests the call profile
- Modem set to 2G/3G (VoLTE doesn’t run yet; this turned out to be irrelevant for the crash)
- The same phone and SIM make calls on Ubuntu Touch without problems
- The only fix I had until now was a full reflash
Timeline
- 1 Sept: fresh flash, calls worked for about six weeks (several successful calls on 17 Oct)
- 19 Oct: black screen (suspected out-of-memory), forced reset with the power button held for 15 s. The first call after that was silent, and every call since then.
- Earlier episodes (for example 28 Aug: 2 h call, display flapping, hwcomposer restart loop, watchdog reboot, afterwards silent calls) fit the same pattern. My suspicion is that the trigger is an unclean reboot, not elapsed time. I only have timestamps proving this for the 19 Oct case.
What the logs show
At call setup the kernel log shows this sequence, every time:
mt6768-audio 11220000.mt6768-afe-pcm: mt6768_afe_runtime_resume()
mt6768-audio 11220000.mt6768-afe-pcm: mtk_pcm_en_event(), name PCM_2_EN, event 0x1
kernel: [pid:pulseaudio] Core dump to |/usr/lib/systemd/systemd-coredump ... 11 ... pipe failed
callaudiod: Error in PulseAudio context: Connection terminated
mt6768-audio ...: mtk_pcm_en_event(), name PCM_2_EN, event 0x8
Signal 11 is a segfault. The crash happens at call setup, not when the other side picks up. Coredumps cannot be written on this kernel (CONFIG_STATIC_USERMODEHELPER_PATH=/system_ext/bin/aee_core_forwarder, hence “pipe failed”), and there is no kernel source or debugfs, so I used gdb and the HAL binary directly.
Side effect of the crash, not the cause: /sys/bus/platform/devices/11220000.mt6768-afe-pcm/power/runtime_status stays active. Five mixer switches are left on (PCM_2_PB_CH1 ADDA_UL_CH1, PCM_2_PB_CH2 ADDA_UL_CH2, ADDA_DL_CH1 PCM_2_CAP_CH1, ADDA_DL_CH2 PCM_2_CAP_CH1, Sidetone Filter Switch) and are saved in /var/lib/alsa/asound.state, so the status survives reboots, poweroff and battery removal. Switching them off gives suspended again, but the next call crashes again and puts everything back. A reproduction started from suspended and clean mixer routes still crashed, so this state is not a precondition.
Getting a usable crash site
gdb procedure and how I mapped addresses
systemctl --user stop pulseaudio.service pulseaudio.socket
XDG_RUNTIME_DIR=/run/user/$(id -u) gdb --args /usr/bin/pulseaudio --daemonize=no --log-target=stderr
(gdb) run
# start a call, wait for SIGSEGV
(gdb) info registers
(gdb) info proc mappings
bt is useless (no CFI in the vendor libraries), but pc lies inside the executable mapping of audio.primary.mt6768.so. For this library the symbol addresses printed by nm -D --defined-only are identical to file offsets, so:
file offset = pc - (start of r-xp mapping) + (file offset of that mapping)
Libraries move between runs (hybris loader), but $pc expressions work in a live session, for example x/215i $pc-0x131a08+0xf9f58 for the function that contains the crash call. Installing binutils allows objdump -d --start-address=... --stop-address=... directly on the .so, no crash needed.
Result, identical in two independent runs on different boots:
- Faulting instruction:
strb w9, [x0], #1
- Function (exported symbol at file offset 0x1319f8, crash is 0x10 bytes in):
android::sph_memcpy(void*, void const*, unsigned int), a byte copy loop
x0 (destination) = 0x80f7f72dff, not mappedx0 = x24 + x25, where x24 is the start of the mapping /dev/ccci_raw_audio (0xd000 bytes, rw-s) and x25 = 0xfffffdffx19 = 0xcd0 (length 3280), x23 = 0xffffffff; only the ASLR-dependent addresses differ between runs
Why the destination is wrong
The caller is android::SpeechExtMemCCCI::shm_region_write_from_linear(region_info_t*, void const*, unsigned int). From the disassembly (field names are my interpretation):
region_info is a 16-byte struct in the shared memory: {offset, size, read_idx, write_idx}; for the failing region it sits at base + 0x28- the destination is
base + offset + write_idx
- the only validation is
size != 0 and idx < size
From the registers, the failing descriptor held {0xfffffdff, 0xffffffff, 0, 0}. That passes both checks and gives the wild pointer. I could not read the memory directly, because gdb answers “Cannot access memory” for this device mapping (probably because ptrace cannot read it).
The correct header is written by SpeechExtMemCCCI::formatShareMemory():
What formatShareMemory() writes (from objdump)
- checks that the mapping exists and is larger than 0xcfff bytes (mine is 0xd000)
- calls
get_uint32_from_mixctrl("vendor.audiohal.speech.shm_init")
- if the value is 0: formats the header, zeroes the data areas, then calls
set_uint32_to_mixctrl("vendor.audiohal.speech.shm_init", 1)
- if the value is not 0: skips formatting entirely and only logs the existing header values
Layout it writes:
| Part |
Offset in mapping |
Size / content |
| guard pattern |
0x00 |
32 bytes of 0x0a |
| flags |
0x20 |
bit 0 set after formatting |
| region 0 descriptor |
0x28 |
{0x80, 0x3000, 0, 0} |
| region 1 descriptor |
0x38 |
{0x3080, 0x2000, 0, 0} |
| region 2 descriptor |
0x48 |
{0x5080, 0x7f60, 0, 0} |
| tail guard |
0xcfe0 |
32 bytes of 0x0a |
The name vendor.audiohal.speech.shm_init matches the ALSA control Speech_SHM_Init (there are also Speech_SHM_USIP and Speech_SHM_Widx). I did not verify the name mapping in the code, but the behaviour fits: Speech_SHM_Init is a boolean mixer control, it is saved in asound.state (value true in mine), and it is on during calls. I also confirmed that PulseAudio itself sets it back to 1 at start-up: stop PulseAudio, cset Speech_SHM_Init 0, cget shows off, start PulseAudio, after 8 s cget shows on.
The workaround
What I ran with PulseAudio running:
for c in 'PCM_2_PB_CH1 ADDA_UL_CH1' 'PCM_2_PB_CH2 ADDA_UL_CH2' \
'ADDA_DL_CH1 PCM_2_CAP_CH1' 'ADDA_DL_CH2 PCM_2_CAP_CH1' \
'Sidetone Filter Switch' 'Speech_SHM_Init'; do
sudo amixer -c 0 cset name="$c" 0 >/dev/null
done
Result: the next call still crashed the old PulseAudio instance, the restarted instance did not crash, audio worked, and after hanging up runtime_status went back to suspended for the first time since hart reset. The cleaner variant is to stop PulseAudio first and start it again afterwards.
To make this happen at every boot before PulseAudio starts, I added this unit (my UID is 32011, check yours with id -u; the wait loop is there because the control may not exist yet):
# /etc/systemd/system/speech-shm-reset.service
[Unit]
Description=Reset Speech_SHM_Init so the audio HAL reformats the shared memory
After=sound.target alsa-restore.service
Before=user@32011.service
[Service]
Type=oneshot
ExecStartPre=/bin/sh -c 'for i in $(seq 1 20); do /usr/bin/amixer -c 0 cget name=Speech_SHM_Init >/dev/null 2>&1 && exit 0; sleep 1; done; exit 0'
ExecStart=/usr/bin/amixer -c 0 cset name=Speech_SHM_Init 0
[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload
sudo systemctl enable speech-shm-reset.service
After a normal reboot the unit ran right after alsa-restore (alsa-restore.service took about 96 s on that boot, cause unknown), and the first call worked without a crash. A healthy system survives normal reboots anyway, so this only proves that the unit does no harm. The real test is a hard reset, which I have not done yet.
Related problem: the black screen that forces the hard resets (and the earlyoom mitigation)
My working theory is that the broken audio state is triggered by an unclean reboot. The unclean reboot was a forced reset after a black screen, so keeping the display stack alive is the first line of defence. The Speech_SHM_Init unit above is the second one.
What I saw
- The screen went black right after opening Firefox (I was away from home, so no SSH). The same morning it had already gone black once, and SSH still worked then. RAM and swap were full, and hanging Firefox processes had to be killed afterwards. The journal shows
phosh segfaulting at 15:50:03, two seconds after Firefox had opened and closed the V4L2 hardware video encoder and decoder (17000000.vcodec_enc, 16000000.vcodec_dec), followed at 15:50:10 by phosh.service: Scheduled restart job and the start of android-service@hwcomposer.service.
- That last line is less alarming than it looks:
android-service@hwcomposer.service has BindsTo=phosh.service and Before=phosh.service. The “hwcomposer crash loop” I first suspected is really phosh restarting.
- Another incident after a long call:
phoc aborted with SIGABRT, phosh logged “Error reading events from display: Connection reset by peer” and died, and the whole session restarted. The proximity sensor was already disabled at that time, so it is not the cause (gsettings set mobi.phosh.shell.proximity enable-proximity-sensor false).
- Second incident: The OOM killer fired (triggered by
kgx). The Mali driver’s OOM notifier listed dev mali0 1287932 kB in total, of which loupe (image viewer) held 1143684 kB of GPU memory.
- During a 2 h call, the display flapped about 130 suspend/resume cycles per minute, followed by a hwcomposer loop and a watchdog reboot.
My interpretation (a guess): GPU memory held by a single app can exhaust the system without being visible in the usual tools. Then the kernel OOM killer or a crashing compositor takes the session down, and in the worst case the phone becomes unresponsive and needs the 15 s power-button reset.
What I changed
- avoiding
loupe and Firefox for large images and video
- Firefox:
media.hardware-video-decoding.enabled set to false in about:config (the V4L2 path had been opened right before the phosh crash)
earlyoom with --prefer for the likely culprits
earlyoom setup
sudo apt install earlyoom
sudoedit /etc/default/earlyoom
# /etc/default/earlyoom
EARLYOOM_ARGS="-m 8 -s 30 --prefer '(loupe|Web Content|Isolated Web Co|firefox)' --avoid '(phosh|phoc|gnome-shell|systemd)'"
sudo systemctl restart earlyoom
systemctl status earlyoom --no-pager
journalctl -u earlyoom -b --no-pager | tail
Notes:
--prefer and --avoid are regular expressions matched against the process name. They matter here because earlyoom chooses its victim by the kernel’s badness score, which is based on RSS and does not include GPU memory. Without --prefer it would probably kill an innocent process while the real hog keeps its gigabyte of GPU memory (my reasoning, not verified).- I could not verify whether GPU allocations lower
MemAvailable early enough for earlyoom to react. A fast leak may still beat it.
- After an OOM event,
journalctl -k | grep -i -B2 -A8 'OOM notifier' shows the per-process GPU numbers from the Mali driver.
Status
If anyone knows how to read per-process Mali memory on a live system without debugfs, or has seen loupe, Firefox or other apps balloon GPU memory on this device, please say so.
What I ruled out
None of these changed the crash:
- Thread priority (
setpriority errno 13 appears in working and broken states alike)
- Contents of
/data/vendor and the PulseAudio .tdb files (stream-volumes.tdb moved away)
- Deleting the NVRAM files
NA87_000 and ER1D_002, and later all 140 files in /mnt/vendor/nvdata/md/NVRAM/NVD_DATA/ (the modem regenerates them with the same sizes, the crash stays)
- Reboot, poweroff, battery removal
- Memory or disk exhaustion
- Package updates:
dpkg.log and apt/history.log show no installs between the last working call and the first failure
- Masking
alsa-restore.service and alsa-state.service, moving asound.state away and rebooting (see open questions, this does not fit my model)
- Modem side:
/sys/kernel/ccci/boot, version and mdsys1/parameter are identical to a capture from a fresh flash, and the kernel ccci messages are identical across boots
- Filesystem damage after the forced reset: ext4 replayed the journal (2 orphan inodes), nothing audio-related was empty or changed
- An Android tombstone of
vendor.mediatek.hardware.mtkpower@1.0-service after the reset (heap corruption abort one second after process start). It came after the first audio failure, so it is not the trigger.
- The install zip contains no modem data (only
boot, dtbo, vbmeta, recovery, userdata and the script)
- Careful with: unbind/bind of the AFE driver,
echo mem > /sys/power/state and writes to /sys/kernel/ccci/md_en all hang the phone (15 s power button needed). I stayed away from the persist, nvdata and nvram partitions, and raw nvram/nvdata images contain IMEI and calibration data.
Open questions
- Why does the broken state survive reboots, and why did my earlier
asound.state test not help? My working model: the flag lives in the mixer state and is restored at boot, while the shared-memory header lives in RAM and is lost after a hard reset, so the HAL believes it is already formatted. That would explain “only a reflash helps” (there is no asound.state after a flash). But a boot with alsa-restore masked and asound.state moved away should then have worked, and it did not. Maybe the kernel default of the control is 1, but I have not checked.
- Does writing the control itself trigger something in the kernel (or on the audio coprocessor)? My boot logs contain
[SCP] scp_send_obj lines. A check of the kernel log around the write was inconclusive because WLAN messages drowned everything.
- Does the unit prevent the failure after a hard reset? If it still breaks despite the unit, the most useful data would be
journalctl -b -u speech-shm-reset.service, then trying separately: only restarting PulseAudio, only cset Speech_SHM_Init 0 with PulseAudio running, and stopping PulseAudio, cset 0, starting it again.
- The last run for six weeks before the forced reset.
How to check whether you have the same problem
journalctl -k | grep -E 'PCM_2_EN|Core dump' # core dump right after event 0x1?
amixer -c 0 cget name='Speech_SHM_Init' # on/off
amixer -c 0 contents | grep -B2 ': values=on' | grep -o "name='[^']*'"
cat /sys/bus/platform/devices/11220000.mt6768-afe-pcm/power/runtime_status
grep -A8 Speech_SHM_Init /var/lib/alsa/asound.state
If you can run the gdb procedure above, compare x23 and x25 at the crash (0xffffffff and 0xfffffdff for me) and the faulting instruction.
For the maintainers
- The HAL’s only plausibility check for the region descriptor is
size == 0, so a stale descriptor leads to a crash instead of an error message.
- Whether the adaptation or the image should reset
Speech_SHM_Init at boot is probably worth a look. As far as I can tell, a missing asound.state (fresh flash) leads to a clean format, a restored true skips it.
- A related but separate problem on my device is the black-screen and phosh-restart issue after long calls (Mali memory,
loupe, Firefox V4L2), which is what forces the hard resets in the first place.
I am happy to post more raw output, the disassembly of formatShareMemory or the exact commands for any step. If anyone has seen a non-reflash recovery or knows what the Speech_SHM_Init write does in the kernel driver, I would love to hear it.