GMSL2 link lock failure for MAX96712

Please provide the following info (tick the boxes after creating this topic):
Software Version
DRIVE OS 6.0.8.1
DRIVE OS 6.0.6
DRIVE OS 6.0.5
DRIVE OS 6.0.4 (rev. 1)
DRIVE OS 6.0.4 SDK
other

Target Operating System
Linux
QNX
other

Hardware Platform
DRIVE AGX Orin Developer Kit (940-63710-0010-300)
DRIVE AGX Orin Developer Kit (940-63710-0010-200)
DRIVE AGX Orin Developer Kit (940-63710-0010-100)
DRIVE AGX Orin Developer Kit (940-63710-0010-D00)
DRIVE AGX Orin Developer Kit (940-63710-0010-C00)
DRIVE AGX Orin Developer Kit (not sure its number)
other

SDK Manager Version
1.9.3.10904
other

Host Machine Version
native Ubuntu Linux 20.04 Host installed with SDK Manager
native Ubuntu Linux 20.04 Host installed with DRIVE OS Docker Containers
native Ubuntu Linux 18.04 Host installed with DRIVE OS Docker Containers
other

Hi,

We are seeing a failure in GMSL2 link lock for the MAX96712 deserializer. During a call to INvSIPLCamera::Init() we see this failure:

Mar 29 13:09:07 tegra-ubuntu: [6545283201] Module_id 30 Severity 2 : devices/MAX96712DeserializerDriver/cdi_max96712.c 2604
Mar 29 13:09:07 tegra-ubuntu: [6545287911] Module_id 30 Severity 2 : MAX96712: GMSL2 link lock not detected on link 0
Mar 29 13:09:07 tegra-ubuntu: [6545288308] Module_id 30 Severity 2 : devices/MAX96712DeserializerDriver/CNvMMax96712.cpp 254
Mar 29 13:09:07 tegra-ubuntu: [6545288459] Module_id 30 Severity 2 : MAX96712: MAX96712SetDefaults failed with NvMedia error 7
Mar 29 13:09:07 tegra-ubuntu: [6545288851] Module_id 30 Severity 2 : src/devblk/common/ddi/CNvMDevice.cpp 71
Mar 29 13:09:07 tegra-ubuntu: [6545289031] Module_id 30 Severity 2 : Driver failed Init() call 10
Mar 29 13:09:07 tegra-ubuntu: [6545289722] Module_id 30 Severity 2 : src/devblk/common/core/CNvMDeviceBlock.cpp 793
Mar 29 13:09:07 tegra-ubuntu: [6545289878] Module_id 30 Severity 2 : Deserializer init failed with SIPL error 10
Mar 29 13:09:07 tegra-ubuntu: [6545290127] Module_id 30 Severity 2 : src/core/CNvMDeviceBlockMgr.cpp 616
Mar 29 13:09:07 tegra-ubuntu: [6545290340] Module_id 30 Severity 2 : Device block init failed. Status: 10
Mar 29 13:09:07 tegra-ubuntu: [6545327257] Module_id 30 Severity 2 : src/core/CNvMDeviceBlockMgr.cpp 244
Mar 29 13:09:07 tegra-ubuntu: [6545341631] Module_id 30 Severity 2 : Device block action failed for action 1
Mar 29 13:09:07 tegra-ubuntu: [6545342160] Module_id 30 Severity 2 : src/core/CNvMDeviceBlockMgr.cpp 246
Mar 29 13:09:07 tegra-ubuntu: [6545342794] Module_id 30 Severity 2 : Device block action failed for block 0
Mar 29 13:09:07 tegra-ubuntu: [6545343043] Module_id 30 Severity 2 : src/core/CNvMDeviceBlockMgr.cpp 278
Mar 29 13:09:07 tegra-ubuntu: [6545343281] Module_id 30 Severity 2 : Failed to initialize device blocks
Mar 29 13:09:07 tegra-ubuntu: [6545343478] Module_id 30 Severity 2 : src/core/CNvMCamera.cpp 1142
Mar 29 13:09:07 tegra-ubuntu: [6545343593] Module_id 30 Severity 2 : DeviceBlock manager init failed
Mar 29 13:09:07 tegra-ubuntu: [6545837372] Module_id 30 Severity 2 : src/core/pipelineMgr/spmgr/pipeline/blocks/CNvFusaISPBlock.cpp 450

I dug a bit into the MAX96712 deserializer driver in the SDK and I see a TODO comment (cdi_max96712.c line 2583) mentioning that the link lock may take >100ms intermittently. Am I correct in thinking that this may have been the root cause of the above failure? Has Nvidia made any progress on figuring out when this may occur and any potential bugfix?

Thank you.

Dear @jmanning2,
May I know the used camera module? Is it reproduced with nvsipl_camera application? Also, could you share complete syslog output related to the application

The camera module we’ve observed this on is the Quanta HZKK-1 / Sony IMX623. The next time it pops up I’ll ensure we can reproduce w/ nvsipl_camera, currently we’ve just been rebooting and it seems to resolve the issue. Below is the full syslog

Apr 01 12:58:51 tegra-ubuntu  [5257459663] Module_id 30 Severity 2 : devices/MAX96712DeserializerDriver/cdi_max96712.c 2604
Apr 01 12:58:51 tegra-ubuntu  [5257464454] Module_id 30 Severity 2 : MAX96712: GMSL2 link lock not detected on link 0
Apr 01 12:58:51 tegra-ubuntu  [5257465052] Module_id 30 Severity 2 : devices/MAX96712DeserializerDriver/CNvMMax96712.cpp 254
Apr 01 12:58:51 tegra-ubuntu  [5257465216] Module_id 30 Severity 2 : MAX96712: MAX96712SetDefaults failed with NvMedia error 7
Apr 01 12:58:51 tegra-ubuntu  [5257465559] Module_id 30 Severity 2 : src/devblk/common/ddi/CNvMDevice.cpp 71
Apr 01 12:58:51 tegra-ubuntu  [5257465723] Module_id 30 Severity 2 : Driver failed Init() call 10
Apr 01 12:58:51 tegra-ubuntu  [5257466563] Module_id 30 Severity 2 : src/devblk/common/core/CNvMDeviceBlock.cpp 793
Apr 01 12:58:51 tegra-ubuntu  [5257466720] Module_id 30 Severity 2 : Deserializer init failed with SIPL error 10
Apr 01 12:58:51 tegra-ubuntu  [5257467027] Module_id 30 Severity 2 : src/core/CNvMDeviceBlockMgr.cpp 616
Apr 01 12:58:51 tegra-ubuntu  [5257467235] Module_id 30 Severity 2 : Device block init failed. Status: 10
Apr 01 12:58:51 tegra-ubuntu  [5257467921] Module_id 30 Severity 2 : src/core/CNvMDeviceBlockMgr.cpp 244
Apr 01 12:58:51 tegra-ubuntu  [5257468646] Module_id 30 Severity 2 : Device block action failed for action 1
Apr 01 12:58:51 tegra-ubuntu  [5257468906] Module_id 30 Severity 2 : src/core/CNvMDeviceBlockMgr.cpp 246
Apr 01 12:58:51 tegra-ubuntu  [5257469361] Module_id 30 Severity 2 : Device block action failed for block 0
Apr 01 12:58:51 tegra-ubuntu  [5257469663] Module_id 30 Severity 2 : src/core/CNvMDeviceBlockMgr.cpp 278
Apr 01 12:58:51 tegra-ubuntu  [5257469932] Module_id 30 Severity 2 : Failed to initialize device blocks
Apr 01 12:58:51 tegra-ubuntu  [5257470140] Module_id 30 Severity 2 : src/core/CNvMCamera.cpp 1142
Apr 01 12:58:51 tegra-ubuntu  [5257470332] Module_id 30 Severity 2 : DeviceBlock manager init failed
Apr 01 12:58:51 tegra-ubuntu  [5257795614] Module_id 30 Severity 2 : src/core/pipelineMgr/spmgr/pipeline/blocks/CNvFusaISPBlock.cpp 450
Apr 01 12:58:51 tegra-ubuntu  [5257798515] Module_id 30 Severity 2 : FusaISP: Skiping Deinit as block was not initialized 1
Apr 01 12:58:51 tegra-ubuntu  [5267229908] Module_id 30 Severity 2 : src/devblk/common/core/CNvMDeviceBlock.cpp 1235
Apr 01 12:58:51 tegra-ubuntu  [5267231855] Module_id 30 Severity 2 : CNvMDeviceBlock must be in the initialized state or the stopped state before attempting to deinitialize
Apr 01 12:58:51 tegra-ubuntu  [5267232216] Module_id 30 Severity 2 : src/core/CNvMDeviceBlockMgr.cpp 638
Apr 01 12:58:51 tegra-ubuntu  [5267232378] Module_id 30 Severity 2 : Device block deinit failed. Status: 6
Apr 01 12:58:51 tegra-ubuntu  [5267233226] Module_id 30 Severity 2 : src/core/CNvMDeviceBlockMgr.cpp 244
Apr 01 12:58:51 tegra-ubuntu  [5267234330] Module_id 30 Severity 2 : Device block action failed for action 4
Apr 01 12:58:51 tegra-ubuntu  [5267234567] Module_id 30 Severity 2 : src/core/CNvMDeviceBlockMgr.cpp 246
Apr 01 12:58:51 tegra-ubuntu  [5267234830] Module_id 30 Severity 2 : Device block action failed for block 0
Apr 01 12:58:51 tegra-ubuntu  [5276615257] Module_id 30 Severity 2 : src/devblk/common/core/CNvMDeviceBlock.cpp 1235
Apr 01 12:58:51 tegra-ubuntu  [5276616740] Module_id 30 Severity 2 : CNvMDeviceBlock must be in the initialized state or the stopped state before attempting to deinitialize

Dear @jmanning2,
Thanks for update.

Hello!

Rebooting is a valid solution in a bench or short term scenario, but it is not a good solution for us in testing. Are there any other indicators we should be checking to see if this can be solved without a power cycle?