Integrating vIDM with NSX ALB (formally known as Avi) is documented officially here: https://avinetworks.com/docs/21.1/configuring-saml-with-workspace-one-for-avi.
I was easily confused by it, due to some of the configuration being masked out. These are the same steps with my own notes & screenshots.