Which is better for an Online Brand? Search Engine Marketing or Social Media Optimization

They both are; SEM is here for long time and it has proven its worth while SMM is relatively new as compared to SEM and a lot of companies are exploring new channels to reach out customers.
Though companies should focus more on SMM as it has a 'viral' factor which SEM will never be able to compete but still, they both are important.
Most importantly, it depends on their target market and audience too!

Interesting question no doubt.
Its an answer that foxes pretty much any digital marketer or asset owner consistently.
IMO the answer lies in your organization's objective, nature of the product/ service & finally as already pointed out where the TG is relatively better represented or influenced.
That being said, SEM's strengths lie in the fact that it assures you presence (providing you pay for it!) in SERPs for keywords that you believe are most relative to your product/ service.
Used smartly and in hygiene with SEO the results can be fantastic.
SMO's strengths lie in the fact that today mostly any TG that's well represented online would be also well represented in popular Social Media platforms.
What then is left to do is develop a deep understanding of what appeals & influences your TG to give the response you want.
I daresay, many a time this might initially come from offline research! Once you have the levers, its far better ROI to pursue them via SMO with some SMM in the mix.

As Saad said, both are important, though I would say the balance would change depending on target audience.
If your audience is heavily present on social media channels, go for that.
If it is more B2B, then SEM will be more useful.

Thank you for sharing your views, But in a country where there is no Law concerning Digital Media, I have seen local brands operating full fledged business online without making any effort concerning SEM or SMO.
The customer ends up searching for their brand but eventually see's list of results which are not owned by the company itself - Hence the perception created at that very particular moment is "AWESOME" but in real it can be something far from reality.
Example of such can be MegaMall

Search Engine Optimization and Social Media Marketing both are very important at the same time , because Google and other Search Engines gives priority to the Website which is active on Social Media bcz social media is connected to your website.
May be some noticed that when you search for your own brand on Google you will find the Facebook Page , Twitter page , Google+ and LinkedIn page on the first page for that specific keyword or brand name.
From Social Media you can drive traffic to your website by posting interested content with link to your website for the detail about the post.
It will also increase time duration on your website of the visitors if you have engaging content .

I would argue and say the term "better" is not the word to be used in the question.
They are both important and effective when it comes to online brands and strategies.
I think they both should be practiced in parallel depending on the situation, audience and campaigns of the brand.

Ofcourse Search Engine Marketing (SEM).
Have a Nice Day.
Syed Waqas Saghir Internet Marketing Expert.
http://www.cointelligence.co/

people are depending more on their social media channels to come up with results and make decisions regarding different brands , since its endorsed and approved by their social circles (word of mouse) moreover if a brand's social media profiles are strong enough they are more likely to appear in the top results in the search engine but SEM cannot be ignored but it comes second to SMM