My cheap cameras includes a Holga 120 GN, Holga 120 Pan and Horizon Perfekt (though this one is not as cheap).
The thing I like about them is that they don't require batteries and they always work that way. They're lightweight so it's easy to carry them on hikes or adventures. Their simplicity forces intuitive and fast shooting.
I started with underwater photography when I was about 15 years old. It had so much impact in my life that I am now an oceanographer.
Tools or equipments have nothing to do with the beauty of your photographs. With 50 bucks you can get a Holga and a roll of 120 film. If you want to invest on something, start by buying a good film scanner.