If you're starting out, keeping costs to a minimum are always a concern, and in that regard people often choose the PayPal route. This works for many.
Personally when I start online stores and other businesses I usually end up investing money rather than going with bare minimum. This means getting a hosting account that supports ecommerce, a dedicated IP, an SSL Certificate, a host that's PCI Compliant, a merchant account for Visa & Mastercard, the DBA & Bank Account, and depending on the operation, Incorporation of the business.
Will PayPal cover the bases off the start - yes, but if you're putting money into your business, going with a merchant account and SSL Certificate etc would be the route I'd pick personally.