Skip to content

[RFC]Suspend/Resume flow for APL - #32

Closed
ranj063 wants to merge 10 commits into
thesofproject:topic/sof-devfrom
ranj063:topic/suspend
Closed

[RFC]Suspend/Resume flow for APL#32
ranj063 wants to merge 10 commits into
thesofproject:topic/sof-devfrom
ranj063:topic/suspend

Conversation

@ranj063

@ranj063 ranj063 commented Jul 13, 2018

Copy link
Copy Markdown
Collaborator

This patchset aims to implement the preliminary flow for suspend/resume for APL.

V2: Now the pipeline status is restored at resume by sending the ipc's to create all the components and set up the pipeline. With this change, the DSP is power off at suspend and when it resumes from suspend, audio playback works normally.

But when suspend is invoked while audio playback is in progress, playback does not resume and aplay quits with the error "pcm_write:2011: write error: Input/output error".

V3: changes made to reflect previous comments

Audio playback resumes normally after suspend/resume.

V4: Added support for runtime PM and made changes based on previous feedback.

Now, both runtime PM and suspend/resume work well.

2 tasks still left to do:

  1. store/restore kcontrol values
  2. free sroute/connect during route_unload.

@ranj063

ranj063 commented Jul 13, 2018

Copy link
Copy Markdown
Collaborator Author

Here's the dmesg log from the point when suspend occurs until after the firmware boots up.
[ 1271.092324] PM: hibernation entry
[ 1271.092743] PM: Syncing filesystems ...
[ 1271.111902] PM: done.
[ 1271.111907] Freezing user space processes ... (elapsed 0.001 seconds) done.
[ 1271.113553] OOM killer disabled.
[ 1271.113686] PM: Marking nosave pages: [mem 0x00000000-0x00000fff]
[ 1271.113688] PM: Marking nosave pages: [mem 0x0003f000-0x0003ffff]
[ 1271.113689] PM: Marking nosave pages: [mem 0x0009e000-0x000fffff]
[ 1271.113692] PM: Marking nosave pages: [mem 0x10000000-0x12150fff]
[ 1271.113822] PM: Marking nosave pages: [mem 0x778a0000-0x778a0fff]
[ 1271.113823] PM: Marking nosave pages: [mem 0x77b12000-0x7a08ffff]
[ 1271.113969] PM: Marking nosave pages: [mem 0x7a3fe000-0x7a428fff]
[ 1271.113970] PM: Marking nosave pages: [mem 0x7a965000-0x7a966fff]
[ 1271.113972] PM: Marking nosave pages: [mem 0x7b000000-0xffffffff]
[ 1271.116164] PM: Basic memory bitmaps created
[ 1271.116239] PM: Preallocating image memory... done (allocated 174133 pages)
[ 1271.259697] PM: Allocated 696532 kbytes in 0.14 seconds (4975.22 MB/s)
[ 1271.259698] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
[ 1271.339787] Suspending console(s) (use no_console_suspend to debug)
[ 1271.371521] sof-audio sof-audio: DSP core(s) enabled? 0 : core_mask 3
[ 1271.492289] PM: hibernation debug: Waiting for 5 seconds.
[ 1276.493320] sof-audio sof-audio: loading firmware
[ 1276.493406] sof-audio sof-audio: booting DSP firmware
[ 1276.493555] usb usb1: root hub lost power or was reset
[ 1276.493559] usb usb2: root hub lost power or was reset
[ 1276.493674] sof-audio sof-audio: unstall/run core: core_mask = 1
[ 1276.493678] sof-audio sof-audio: DSP core(s) enabled? 1 : core_mask 1
[ 1276.550947] sof-audio sof-audio: pstream 0 status 0x4
[ 1276.572308] sof-audio sof-audio: ipc: DSP is ready 0x70000000 offset 0x81000
[ 1276.572344] sof-audio sof-audio: Firmware info: version 1.1-bf14b build 32 on Jul 5 2018:22:52:51
[ 1276.572428] sof-audio sof-audio: found ext header type 1 size 0x9c
[ 1276.572451] sof-audio sof-audio: cannot create debugfs entry.
[ 1276.572455] sof-audio sof-audio: cannot create debugfs entry.
[ 1276.572458] sof-audio sof-audio: cannot create debugfs entry.
[ 1276.572462] sof-audio sof-audio: cannot create debugfs entry.
[ 1276.572465] sof-audio sof-audio: cannot create debugfs entry.
[ 1276.572469] sof-audio sof-audio: cannot create debugfs entry.
[ 1276.572472] sof-audio sof-audio: cannot create debugfs entry.
[ 1276.572475] sof-audio sof-audio: mailbox upstream 0x81000 - size 0x1000
[ 1276.572478] sof-audio sof-audio: mailbox downstream 0xa0000 - size 0x2000
[ 1276.572480] sof-audio sof-audio: stream region 0xc1000 - size 0x1000
[ 1276.572482] sof-audio sof-audio: booting DSP firmware completed
[ 1276.572486] sof-audio sof-audio: ipc rx: 0x70000000 done
[ 1276.575709] sof-audio sof-audio: Firmware download successful, booting...
[ 1276.575714] sof-audio sof-audio: firmware boot complete
[ 1276.575926] rtc_cmos 00:02: Alarms can be up to one month in the future
[ 1276.602266] r8169 0000:03:00.0 enp3s0: link down
[ 1276.604724] r8169 0000:02:00.0 enp2s0: link down
[ 1276.814309] ata2: SATA link down (SStatus 4 SControl 300)
[ 1276.818299] ata1: SATA link down (SStatus 4 SControl 300)
[ 1276.848119] usb 1-2: reset high-speed USB device number 2 using xhci_hcd
[ 1277.312140] usb 1-2.3: reset full-speed USB device number 3 using xhci_hcd
[ 1277.562286] PM: Basic memory bitmaps freed
[ 1277.562289] OOM killer enabled.
[ 1277.562290] Restarting tasks ... done.
[ 1277.564141] video LNXVIDEO:00: Restoring backlight state
[ 1277.564144] PM: hibernation exit
[ 1277.658120] IPv6: ADDRCONF(NETDEV_UP): enp2s0: link is not ready
[ 1278.173462] r8169 0000:02:00.0 enp2s0: link up
[ 1278.173500] IPv6: ADDRCONF(NETDEV_CHANGE): enp2s0: link becomes ready
[ 1280.138162] pcm512x i2c-104C5122:00: No SCLK, using BCLK: -2
[ 1280.138181] sof-audio sof-audio: pcm: open stream 0 dir 0
[ 1280.138185] sof-audio sof-audio: period min 192 max 16384 bytes
[ 1280.138188] sof-audio sof-audio: period count 2 max 16
[ 1280.138190] sof-audio sof-audio: buffer max 65536 bytes
[ 1280.138540] sof-audio sof-audio: rate_min: 48000 rate_max: 48000
[ 1280.138544] sof-audio sof-audio: channels_min: 2 channels_max: 2
[ 1280.138547] sof-audio sof-audio: rate_min: 48000 rate_max: 48000
[ 1280.138550] sof-audio sof-audio: channels_min: 2 channels_max: 2
[ 1280.138555] sof-audio sof-audio: rate_min: 48000 rate_max: 48000
[ 1280.138557] sof-audio sof-audio: channels_min: 2 channels_max: 2
[ 1280.138562] sof-audio sof-audio: pcm: hw params stream 0 dir 0
[ 1280.138566] sof-audio sof-audio: generating page table for 00000000575ba841 size 0xff00 pages 16
[ 1280.138570] sof-audio sof-audio: pfn i 0 idx 0 pfn 1795a0
[ 1280.138573] sof-audio sof-audio: pfn i 1 idx 2 pfn 1795a1
[ 1280.138575] sof-audio sof-audio: pfn i 2 idx 5 pfn 1795a2
[ 1280.138578] sof-audio sof-audio: pfn i 3 idx 7 pfn 1795a3
[ 1280.138581] sof-audio sof-audio: pfn i 4 idx 10 pfn 1795a4
[ 1280.138583] sof-audio sof-audio: pfn i 5 idx 12 pfn 1795a5
[ 1280.138586] sof-audio sof-audio: pfn i 6 idx 15 pfn 1795a6
[ 1280.138588] sof-audio sof-audio: pfn i 7 idx 17 pfn 1795a7
[ 1280.138591] sof-audio sof-audio: pfn i 8 idx 20 pfn 1795a8
[ 1280.138629] sof-audio sof-audio: pfn i 9 idx 22 pfn 1795a9
[ 1280.138632] sof-audio sof-audio: pfn i 10 idx 25 pfn 1795aa
[ 1280.138635] sof-audio sof-audio: pfn i 11 idx 27 pfn 1795ab
[ 1280.138637] sof-audio sof-audio: pfn i 12 idx 30 pfn 1795ac
[ 1280.138661] sof-audio sof-audio: pfn i 13 idx 32 pfn 1795ad
[ 1280.138664] sof-audio sof-audio: pfn i 14 idx 35 pfn 1795ae
[ 1280.138666] sof-audio sof-audio: pfn i 15 idx 37 pfn 1795af
[ 1280.138686] sof-audio sof-audio: period_bytes:0x3f00
[ 1280.138694] sof-audio sof-audio: stream_tag 1
[ 1280.138725] sof-audio sof-audio: ipc: send 0x60010000
[ 1280.138824] sof-audio sof-audio: error: ipc error for 0x60010000 size 0x14
[ 1280.146584] sof-audio sof-audio: ASoC: sof-audio hw params failed: -19
[ 1280.153897] Passthrough: ASoC: hw_params FE failed -19
[ 1280.153935] sof-audio sof-audio: pcm: free stream 0 dir 0
[ 1280.159819] sof-audio sof-audio: ipc: send 0x60030000
[ 1280.159873] sof-audio sof-audio: error: ipc error for 0x60030000 size 0xc
[ 1280.167679] sof-audio sof-audio: pcm: close stream 0 dir 0

@RanderWang

RanderWang commented Jul 13, 2018

Copy link
Copy Markdown

From kernel log, rmbox should be not work ? so no rmbox message?
for 0x60010000 error, I think it is caused by no topology. So enable rmbox is important for debug.

And I think each internal buffer pointer or component status should be kept ?Or OS would send hw_param and start again?

@RanderWang

Copy link
Copy Markdown

What happens to all the components created when the topology is loaded before suspend?
[Rander] Each component initialize its status

Should the topology be reloaded after resume?
[Rander] Each component should restore status. loading topology would restore each components to initialized status, maybe extra restore?

@RanderWang

Copy link
Copy Markdown

I am not sure: some memory in DSP is still valid, so it can be used to keep some status

@plbossart

plbossart commented Jul 13, 2018

Copy link
Copy Markdown
Member

@ranj063 : I don't think the topology should be parsed again on resume, but certainly anything that results in an IPC should be invoked on resume, e.g. SSP settings and pipeline commands (likely why the hw_params fail). I am afraid today we combined token parsing and IPC, maybe not such a good idea in hindsight.

@lgirdwood

Copy link
Copy Markdown
Member

@ranj063 we probably dont want to pasre the topology again, but sof/topology.c does track a lot of the topology structures locally when probing. i.e. we have a list of widgets, controls etc. We should store a pointer to the topology raw structure for each topology object and on resume we iterate through all the topology object lists and send IPCs.

@ranj063

ranj063 commented Jul 16, 2018

Copy link
Copy Markdown
Collaborator Author

@plbossart @lgirdwood Thanks! Let me re-send the ipc's to restore the pipeline from the topology objects stored in the driver.

@ranj063

ranj063 commented Jul 17, 2018

Copy link
Copy Markdown
Collaborator Author

@plbossart @lgirdwood , I have updated the commits in this pull request to restore the pipeline at resume.
With this change, I am able to start playback after resume. But I still fail when I suspend while audio playback is in progress with the error: "pcm_write:2011: write error: Input/output error".

@lgirdwood

Copy link
Copy Markdown
Member

@ranj063 ok, half way there now. Have a look at the legacy baytrail driver (sound/soc/intel/baytral/*) as it stores PCM playback state prior to S3 and restores at S0. You probably need to duplicate this flow.

Comment thread sound/soc/sof/pm.c Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about if we have 2 or more pipelines ? If this restores all pipelines then maybe rename func.

Comment thread sound/soc/sof/pm.c Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just wondering if we need all these case statements given all the IPC data uses a standard header (which includes cmd and size). i.e. we cas the private data to the header type to get size and cmd.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lgirdwood I started it that way but I kept running into errors with it. The only way I can send the ipc without errors is by having all these case statements.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, interesting, what errors ? The only thing the case statements do is cast types

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lgirdwood. I've fixed it now. The error I was seeing was because of not casting the comp objects to (void *) while storing them.

Comment thread sound/soc/sof/intel/hda-dsp.c Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Best to try and send this today as it allows DSP to enter D3 ready state. Any context can be saved later.

Comment thread sound/soc/sof/intel/hda-dsp.c Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd expect most of this function to be in pm.c as generic code. The HW specific part hda_dsp_core_reset_power_down() can be an HW abstracted operation (add ops to the operations callback structure). e.g. we could have snd_sof_dsp_suspend(sdev), snd_sof_dsp_resume(sdev)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree with Liam, this feels too APL-specific.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@plbossart @lgirdwood I have updated this to use the chip cores_mask to make it generic.

@ranj063

ranj063 commented Jul 18, 2018

Copy link
Copy Markdown
Collaborator Author

@lgirdwood I've just pushed the v3 changes. Now I can resume audio playback normally after suspend/resume.

There's an occasional xrun that happens at resume but its been hard to reproduce.

@lgirdwood lgirdwood left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly minor things, glad its working now :)

Comment thread sound/soc/sof/intel/hda-dsp.c Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Btw, we need to be able to handle different core masks, e.g. APL has 2, CNL has 4.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lgirdwood sure. I have tested this to work only for APL as yet. I will make the changes for CNL next.

Comment thread sound/soc/sof/pm.c Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, interesting, what errors ? The only thing the case statements do is cast types

Comment thread sound/soc/sof/pm.c Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd make the stream suspend a separate function

Comment thread sound/soc/sof/pcm.c Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where do we send the IPC start message if we set cmd here ?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lgirdwood It is sent right outside the switch/case block where we set the cmd.

@plbossart plbossart left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good overall, still some style issues and my usual quixotic chase of memory leaks. One functional part also to align times with the firmware so that on resume the trace logs aren't completely misleading.

Comment thread sound/soc/sof/topology.c Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the volume case, don't you need to free swidget->private?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@plbossart good catch. I'll fix it.

Comment thread sound/soc/sof/topology.c Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need to free sroute first? this doesn't look consistent error management, or do you assume that all frees are done in an sof_route_unload()?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you need a more consistent memory allocation/free. Here you free on one error, but not for all the other cases in this function below (look for all the return -EINVAL cases). You should have a set of gotos and deal with errors in a more organized way, or you deal with all the frees in another cleanup routine called on failure.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh yes. let me address this. seems like I've missed the route_load case to fix the memory allocation/free.

Comment thread sound/soc/sof/topology.c Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

well, duh. Can you please update this one, this will create issues with module load/unload and break CI.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@plbossart sure, I am working on it. It wasnt very straightforward which route is being unloaded from the arguments. I need to study this a bit more to implement it correctly.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@plbossart @lgirdwood it doesnt look like route_unload ever gets called.
I see a comment in soc-topology.c in the remove_widget() function that suggests that routes must
be removed before the widget itself is removed. But I cant find any references to route_unload.

Also, the arguments to the route_unload() method seem ambiguous. There's no dobj member in snd_soc_dapm_route structure. So I'm not sure how to get a handle to it in order to be able to remove it. Can you please help?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You will need to deep dive this more as I'm short of time and may also need to modify the API if needed to pass in the extra info.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like we still have a dependency on a core cleanup?

Comment thread sound/soc/sof/pm.c Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need explicit /* fallthrough */ to make tools happy that this is not a programming error

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the reminder. I'll add this where appropriate.

Comment thread sound/soc/sof/pm.c Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ret is not tested in most cases?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@plbossart ret is tested right outside the switch/case block to make sure none of the ipc's failed.

Comment thread sound/soc/sof/pcm.c Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this be squashed?

Comment thread sound/soc/sof/pcm.c Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't there be some IPC sent on suspend, so that e.g. if there is any sort of context saving at the firmware level they'd know about it?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@plbossart I do send the CTX_SAVE ipc during suspend in the suspend callback but not here as it is not a stream ipc message.

Comment thread sound/soc/sof/pcm.c Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW on resume we'd need to pass the new wall clock time to the firmware so that any trace logs are timestamped with the right values.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@plbossart good point. Let me add it if we're not doing that already.

@ranj063

ranj063 commented Jul 20, 2018

Copy link
Copy Markdown
Collaborator Author

@plbossart @lgirdwood I've made some more changes to the flow now.
But the good news is that both suspend/resume and runtime_pm work well. Could you please help review?
This needs some more extensive testing and there are 2 opens to address that I've mentioned in first comment that shows the progression of patches.

@lgirdwood lgirdwood left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry could not see the opens you mentioned ?

Comment thread sound/soc/sof/pm.c Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would also be good to see the widget type failing here swidget->id, we would then have the type and id.

@ranj063

ranj063 commented Jul 20, 2018

Copy link
Copy Markdown
Collaborator Author

@lgirdwood the two opens left are:
2 tasks still left to do:

  1. store/restore kcontrol values
  2. free sroute/connect during route_unload

@plbossart

Copy link
Copy Markdown
Member

@ranj063 do you want us to review/test now or will the two opens come shortly enough that we want to wait for the update.

@ranj063

ranj063 commented Jul 20, 2018

Copy link
Copy Markdown
Collaborator Author

@plbossart I should be done with the 2 opens by today and I think it will be better to test after those.

But I've made some change regarding which device we register the PM callbacks for. I could use some feedback to make sure I've done the right thing.

@plbossart

Copy link
Copy Markdown
Member

@ranj063 can you describe the device change so that the rest of us don't have to reverse-engineer the deltas between patches?

@ranj063

ranj063 commented Jul 20, 2018

Copy link
Copy Markdown
Collaborator Author

@plbossart this is the commit id for the device change in this patchset:
0e6a2a584473a05efbb34185272e1c34cf436fcc

With the callbacks set for the pci device, they never get called as the runtime_pm_enable() is called for the platform device and not the pci device.

So setting the callbacks for the platform device fixes the issue.

@lgirdwood

Copy link
Copy Markdown
Member

@ranj063 for the kcontrol restore, iirc baytrail/haswell drivers did this by caching the values in the host.

@ranj063

ranj063 commented Jul 21, 2018

Copy link
Copy Markdown
Collaborator Author

@lgirdwood, sure. I think I have an idea of how to do this. I will update the PR after I fix it.

@ranj063

ranj063 commented Jul 21, 2018

Copy link
Copy Markdown
Collaborator Author

@plbossart @lgirdwood I've addressed all opens now. Could I request you to please review the latest version.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants