How we can secure data inside an Oracle database so that even DBA cannot see the application data?

You need to deploy Oracle Database Vault

We can deploy separation of duties through Oracle Database Vault and create realms as per your data domains. 

Data in transfer can be secured by using encryption provided by Oracle Advance Security option of Enterprise Edition.

Data at rest can be secured by using Data Masking and subsetting pack

There may be many aspect to this question.

As far as security point of view there are algorithms like 'AES128' or AES264 which privide more comression amd add a level a security to the Data.

Answer to this question lies with the  ORacle Vault.

It is a technique through which even a system administrator who has the control of the whole system would not be able to see the data without the key supplied by the Oracle Vault.

The ORacle vaults needs to open and running then only you would be able to access the data which would have a meaning. 

This can be achieved through

Transparent Tablespace Encryption

by data vault

but even there is no one have the control of product only DBA can Enable Or disable this feature 

Thanks for this update.

Do you have any update whether this kind of feature is even available for SQL Server / Sybase as well?

By using Oracle Database Vault