Hello,

In the documentation is says:

In most cases, you will need to provide the main source password (which you configured in the Station Details and Time section of the Basic Usage guide) to your DJs to allow them to connect to the server with a live source for their broadcasts.

(from http://www.centova.com/doc/cast/user_manual/01_usage_guide/02_advanced_usage/03_Providing_Access_to_DJs )

When DJs have the main source password they don't need their login and password.

Is there any way to limit DJs access while using Icecast?

You could create a mount point for each DJ and config Fallback from one to another.
Like chaining them together, for the last mount in the chain define fallback to /stream and that's it.