Skip to content

check-sof-logger: catch ERROR in firmware traces and fail - #799

Merged
marc-hb merged 1 commit into
thesofproject:mainfrom
marc-hb:catch-logger-error
Jul 28, 2022
Merged

check-sof-logger: catch ERROR in firmware traces and fail#799
marc-hb merged 1 commit into
thesofproject:mainfrom
marc-hb:catch-logger-error

Conversation

@marc-hb

@marc-hb marc-hb commented Nov 1, 2021

Copy link
Copy Markdown
Collaborator

Add function find_logger_error() in case-lib so it can re-used by other
tests later.

Signed-off-by: Marc Herbert marc.herbert@intel.com

@marc-hb

marc-hb commented Nov 2, 2021

Copy link
Copy Markdown
Collaborator Author

https://sof-ci.01.org/softestpr/PR799/build917/devicetest/?model=BDW_WSB_RT286&testcase=check-sof-logger has a very interesting:

 TIMESTAMP      (us)           DELTA  C# COMPONENT          LOCATION                      CONTENT	ktime=117.383s  @  2021-11-01 23:27:33 +0000 UTC
[  1852324251.812] (           0.000) c0 dma-trace             src/trace/dma-trace.c:422  INFO SHM: FW ABI 0x3014000 DBG ABI 0x5003000 tag v1.9-rc1-211-gb0bb21dc0ee6 src hash 0xa9c07289 (ldc hash 0xa9c07289)
[      324686.862] (      324686.875) c0 dma-trace             src/trace/dma-trace.c:605  ERROR dtrace_add_event(): number of dropped logs = 1

@marc-hb

marc-hb commented May 2, 2022

Copy link
Copy Markdown
Collaborator Author

SOFCI TEST

@marc-hb
marc-hb force-pushed the catch-logger-error branch from 9a8bab5 to 10ec30e Compare May 3, 2022 01:09
@marc-hb
marc-hb requested a review from ujfalusi May 3, 2022 01:31
@marc-hb

marc-hb commented May 3, 2022

Copy link
Copy Markdown
Collaborator Author

https://sof-ci.01.org/softestpr/PR799/build917/devicetest/?model=BDW_WSB_RT286&testcase=check-sof-logger has a very interesting:

https://sof-ci.01.org/softestpr/PR799/build16/devicetest has something similar on BYT (BDW is not in PR testing anywmore)

[   178559514.207] (           0.000) c0 dma-trace             src/trace/dma-trace.c:347  INFO SHM: FW ABI 0x3016000 DBG ABI 0x5003000 tags SOF:v2.0-rc1-813-g5cfc37040bbe src hash 0xb72c3013 (ldc hash 0xb72c3013)
[       27442.082] (       27442.082) c0 dma-trace             src/trace/dma-trace.c:654  ERROR dtrace_add_event(): number of dropped logs = 8

@marc-hb
marc-hb force-pushed the catch-logger-error branch from 10ec30e to 31640e2 Compare May 26, 2022 04:21
@marc-hb
marc-hb marked this pull request as ready for review May 26, 2022 05:05
@marc-hb
marc-hb requested a review from a team as a code owner May 26, 2022 05:05
@ujfalusi

Copy link
Copy Markdown
Contributor

@marc-hb, is it a recent issue or it has been there?
It is very likely to happen if the firmware is generating 'enough' trace messages up to the point when sof-logger starts and opens the sof/trace debugfs file to start consuming the dtrace messages.

If you do not start the sof-logger at all then yeah, you are going to drop messages, but is this an error?

@marc-hb

marc-hb commented May 27, 2022

Copy link
Copy Markdown
Collaborator Author

Thanks @ujfalusi for the prompt review!

Interesting. So this reproduced systematically on BYT and BDW... because the buffer is smaller / too small? Or maybe because they have more interfaces and more to log.

So I can just downgrade this ERROR to a WARN, right?

@marc-hb
marc-hb marked this pull request as draft May 27, 2022 18:50
@ujfalusi

Copy link
Copy Markdown
Contributor

@marc-hb, I think it is WARN rather an ERR, yes (if you refer to the fw print about the dropped new entries. It only takes to not start sof-logger to eventually have them, which is fine?

It can explain 'blackouts' in logs

@marc-hb
marc-hb force-pushed the catch-logger-error branch from 31640e2 to c572179 Compare July 27, 2022 01:02
@marc-hb

marc-hb commented Jul 27, 2022

Copy link
Copy Markdown
Collaborator Author

https://sof-ci.01.org/softestpr/PR799/build112/devicetest is all green. I recommend we merge this "as is" and if this error message become too frequent we can downgrade it.

@marc-hb
marc-hb marked this pull request as ready for review July 27, 2022 03:53
Comment thread test-case/check-sof-logger.sh
Add function check_logger_error() in case-lib so it can re-used by other
tests later.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
@marc-hb
marc-hb force-pushed the catch-logger-error branch from c572179 to 87aeb4d Compare July 27, 2022 23:28
@marc-hb

marc-hb commented Jul 28, 2022

Copy link
Copy Markdown
Collaborator Author

https://sof-ci.01.org/softestpr/PR799/build113/devicetest/ has one usual WHL UPX suspend/resume TIMEOUT, everything else is green.

@keqiaozhang keqiaozhang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@marc-hb

marc-hb commented Jul 28, 2022

Copy link
Copy Markdown
Collaborator Author

Thanks @keqiaozhang

@marc-hb
marc-hb merged commit 6544bc1 into thesofproject:main Jul 28, 2022
@marc-hb
marc-hb deleted the catch-logger-error branch July 28, 2022 02:18
@marc-hb

marc-hb commented Jul 29, 2022

Copy link
Copy Markdown
Collaborator Author

Pattern matching fix submitted in

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:logs Log and results collection, storage, etc. False Pass / green failure

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants